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'INTRODUCTION 

This is the final report of an investigation of further evalu- 
ation alternatives for the social studies curriculMm Man: A Course of 
Stucly.* The purpose of the study was not to specify an approach. But to 
clarify alternatives and bases for choosing among them. As a preliminary 
step in developing suggestions, field visits were made to seven MACOS 
Regional Centers.** and to severarschool districts and schools in each 
region, to obtain information about conditions of use of the curriculum. 
Tho obsprvationr. mods and information and '.impress ions obtainod entered 
stronoi> inLo our idCuS about further cvaluauon needs and strategics. 

In tiic cour:.o of Hft--pdriny this report^ we found ourselves re- 
thinking most of our preconceptions and assumptions about further 
evaluation of the MACOS ^package . especially summative evaluation. The 
question of feasibility of alternative evaluation approaches becan^ integ- 
rally enmeshed with the prior question: evaluation to what end, for what 
purpose? The reader will find that we tried consistently to address this 
question on a gross level, but that we have not been able to provide a 
clear analytic framework for answering it. 

The Structure of this report is as follows. In Section I the 



This report was prepared under NSF Grant No. W005707. 
One center had recently been terminated. 



task-of the stu4y and a general view of the processes of curriculum devel- 
opinent projects are described. Section II summarizes the general methods 
employed in this study. Section III gives a brief sunmary outline of 
"findings" from field visits. In Section IV we present further examples 
and discussion of findings, and Section V provides a summary list of char- 
acteristics of the curri culum that appeared important or significant to 
-various respondents in the field. In Section VI we then discuss six 
categories of evaluation questions or concerns related to MACOS. In most 
categories we note characteristics of the curriculum that^seem especially 
germane to the design of further evaluation studies. We then synthesize in 
the next section the analyses made in Section VI into three major kinds 
of further cvnluntion that minht be undnrtahnn. In Section VII we disctir.s 

tive. C.11U sysiPi-nr nv^^v^rr. or.s thst cculd be liPJcrtak.^n. l.'s^ finally 
provide a sun^nary stateiniMit, in Section VIII, of Usues . alternatives and 

recoiruiendati ons . ' i • 

It will be apparent' to the reader that we found our task complex. 
l4\ an effort to avoid vigorous simplistic suggestions we constantly risked 
paralysis from attention to a multitude of issues, uncertainties and possib 
ities. The net "result of efforts to achieve a balance will undoubtedly 
strike many as erring in one direction or the other. For ourselves, we 
believe that erring in the direction of complexity is the better, if 
aesthetically less satisfying, error. 
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The National Science Foundation expressed Its interest in funding, 
a feasibi'lity study to determine what some possible next steps, in terms 
of continuing evaVuation of MACOS. might be. A thorough and excellent* 
formative evaluation had been completed."' Discussions with NSF did not 
provide us with any clear answers to the question, "What are the NSF's 
further interests or goals with MACOS?" Therefore, we suggested that we 
try to depict some alternative strategies for continued evaluation based 
on what we could determine to be the needs, present conditions, and 

c o .,...t,io;-;; ^unf^ tu itiiiiu i ii«ifed j d iv . WHat HHs happenedi 



with the development and dissemination of the curriculum to date? What 
is the present state of its utilization? How effective is MACOS compared 
with other social studies curricula? And so..on. We felt that by talking 
with .lassroom teachers, social studies curriculum specialists within 
school systems, administrators, regional directors and disseminators, and 
members of the Education Development Center, we coulii help focus on 
these and other questions. We hoped such an approach might help outline- 
some priori'ties for further study. In an effort to do this we proposed 
a series of interviews in each of the regions established for- dissemination 
of MACOS. Using the data gathered we shall, in this report, suggest some 
evaluation issues and strategies for NSF's consideration. 
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A. review of'MACOS materials, especially the most recent Seminars 
for Teachers, ^ helped to clarify the potential multiple functions the new 
curriculum might have. It seemed that a consideration of evaluation issues 
wlth'r^pect to the various possible functions of the curriculum might 
provide a way of organizing and integrating the concerns of en^rging 
educational evaluation theory with .the realities of MACOS research, 
development, and utilization. Our hope then, was tliat 'we. coul d clarify, 
at least presumptively, the manifold number of issues involved in MACOS 
as well as suggesting some priorities for further study. 

There are four broad functions a new curriculum may perform: 

1. Education of children 

2. Dr-volorn'-int and frih^nrem.^nt of ixninonnicnl skill'', and / 

3. ' r.h;infif^ in i nc ti 1 utinnr. . . 

',4 ' ■ 

4. Training and development of prospective teachers 

Thei-^ is nothing new about these areas of effect, impact, or concern. Most 

of them are discussed in one form or adBthcr^ for example, by Grobman. 

'^"^ ■ 

Patently, the purpose of a new curriculum such aS MACOS is to 
improve the educational growth and development of students. Equally obvious 
the educational enterprise Is a manifold affair, fraught with uncertainties 
about what causes what, Mr what maximizes what. Undef such circumstances, 
there are legitimately -different strategies and options for improvement. 
Furthermore, the same device may in fact bear, on different dimensions of 
the educational enterprise, depending on how and where it is used. A 

♦ a. . . 



'2 Mr-p- A ^^^i^r^t-j^l^-'l^y- Seminars for Teachers'. Education Development , 
Center^; Inc. , 1970. 

3, Grobman, Hulda. ^^-llijjjjiinJL^Alvit^ Curriculum Projecti. ^ERA Mono 

graph Series of CurriaTlum Evaluation, No. 2. Chicago: Rand McNally, 196^. 



' curriculum package^, then, may peKorm some or all of the functions ^ 
listed above. It 1s"a matter of considerable Interest to try to explicate 
rules or propositions for which ones, when, where, how, etc. 

If one is going to design a new curriculum, he obviously has the 
intention of producing §onsth1ng that will have one or more of the'functrons 
suggested above. But between the design ihtention and 1 ts' accompli shmant 
of one or more of" the curriculum functions are a, number of linking or enabling- 

functions. E.g., 

1. Development- and testing 

2. Dissemination ' 

3. Preparation and installation^ . 

4. Implementation- . . - • 

5. !1a'lnt.on?hf.o Eind support 

7. I'nuitnrinn niiH "v^"! 'J?,ti o;i^ 
Suppose it is the designer's intention to produce a curriculum package that 
will perform particular educational functions for or with elementary school 
children (in the case of MACOS . for example, to develop hypothesis forming 
skills, to speak glibly for the moment-) The design and development of 
certain materials and procedures constitutes a transformation of that intention 
into a mechanism or apparatus for performing that specific educational' 
function. Tests of the«,curriculum are used to determine whether, and 
presumably under what condi tions ,the package does perform it. 



^ Preparation here includes orientation and training of teachers and others 
nrior to or at the start of the curriculiim; installation includes purchasing, 
schedulinq. etc. Jointly they designate the activities necessary or undertaken 
>^ithin a school or school system to put the new curriculum into effect. 
^ This should be a parallel function, not a serial or partially serial 
one like the others. 
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One 6 Valuation issue at this points ccMficern^ the fderitification of 
Intended fuitctions, the4r performance characteristics, and the limiting 
assumptions concerning the performance characteristics (e.g., what skills . ; 
of teachers are assumed? what' kinds of students? wh^t organizational 
arranqemants in schools? etc.}. 'a second eva>u?ti"oh i^sue.^t this point 
concerns the design and development method or procedure, l^at did -ft 
take to accomplish the transformations? What are the implications for 
further curriculum project funding decisions? Can desigh and development 
components be related to costs in^d gmieralizabTe^ sense? 

The curriculum or mechanism intended to generate the designated 
function is then <iisseminated, that is, brought to the attention of 
prqspectivn uscts. \n thi:, tr;msform:,1:ion the intended function m^y he 
rctainsu, luoJirled or. lo:U and/ur uUi^r luucticns an^i pulci.flal tunctior.s ^ 
added. S0i.-.c r.ui.i.c, uT vro.p^aive users will adopi^ tne new curriculum / 
over a given period of time. . Processes .of .. preparation, installation, and 
eventually *implGmsntati on thus, commance. Evaluation issues to that point 

include: • ' 

1. what are the operating characteristics of different dissemination 

mechanisms? are some more efficient than others? how? with what kinds, of 
users? with respect to v/hat kinds of functions? 
• • ' 2. is there a relationship between different kinds of ^dissemination 



^ We are trying here to get at the problem of evaluating what kinds of ^ 
/ approaches, with what kinds of variations . yield what '<^"^5 °f Products 

(?Srriculum packages, for example) under ^'^»^^\^°"ditTons We think this sort 
of analysis is related^to further decision-making about the need for puoiic 
support of curriculum design and developnnsnt. 



id * 
ERIC ... 0 ' .4- 



mechanisms and the further trans fomati on of Intended functlonsT^ 

The new surriculum Is adopted and impleitented, tentatively or . 
permanently. At this staae there is the final set of t.ransforn<at1ons ' . 
leading to resultant functions." Hi tK 'Respect to the; function jlealing with 
education of ..children i ttie evaluatipn Issues center around the traditional 
probieMS of ^asuring effectiveness . or aiore specifically, differential 
effectiveness. That isf'who learns what, under wl^at conditions? Are 
there unexpected results? Are the designer's intended functions realized? 
Hhat support conditions turn o«t to be needed? What further nodifi-cations 

are appropH ate? And so on. 

This highly siniplistie discussion is intended only tO ?uggesf 

that:- 

1. there \ir. dlHen.nt ev^Ouutiuh issues iliffarcnt stages of 

2 the issues arc separable Wl th .respect, to certain decision prob- 
Tens for certain agencies; but data bearing on issues at one point m need 
' to be carried forward to apply at another point; 

3. the' transformations that may occur are only partly predictable, 
though eventually tW are completely (potentially) .knowable. 
We shall .•eturn to these themes. They, are much broader than the con'cept of 
su^ative'evaluation. This brief discussion of an overall process indicates 
that in approaching our task we have tHed to consider a broad spectrum of 
congruences,, contingencies, and decisions. 
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DESCRIPTION OF ESD's PROCEQJRES f. 

« 

Our first steps were to make ourselves more familiar with the , , 
mcOS package and it^ history, We'di^ this through consultation with 
members of the Innovation Teaman the District of Columbia who had been ^ 
teacher?, disseminatoi^ , and trainers for MACOS;' through the review of the 
curriculum materials and dbcunsnts about the^'pnogram; and through discussion 

'''' " • " ' f 

with the f'iACOS evaluation staff at Education Developirenf Center. We then 

prepared questionnaires to be answered in interviews wi ^regional center 

directors and people associated with regional .centers > The e^uestionnal res 

f. . . . 

were also to be used with school administrators, curriculum supervisors, .and 

ieactwiJS. The basic" 1nle«rions of the questions were t© provicte iftforjrwtian 

abbtft What oOservfens saiw the* purposes, both actual and potential, of. 

WtfOSi What n^t»KJas were'.beina u&ei to assess or evaluate the programi wb.ttt 

was being dope to disseminate' IWe program, install it, and support it; 

and y*Ut seemed to be'some of the crucial factors involved in each of these 

activities. '. . ' ' 

"The Educational Studies Dspartment Team visited seven regional 
'centers, twenty-four school districts, and thirty-six schools within, districts. 
School districts visited were splected in discussion with center directors, ■ 
and -final choice was based on interest and practicality. The schools in 
no sense- represent a sample, but it is perfectly clear to the Team that 
they covered a wide variety gf environments and educational conditions. 
We were able "to compose a detaij^^d compendium of responses to various questions 
and thus to identify potentially important variables, based either on direct 
statements gleaned from the talks we had with many people, or on impressions 
derived from the. overall discussions. In -addi tion^we have done an extensive 



perusal .of tRe evaluation literature in general, and of literature ^ 
concerned with social studies and educational change In particular. 

In Appendix A there is a table summarizing the number and characteristics 
of teachers interviewed, and numbers of locations visited. • 
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BRIEF SUMMARY OF INFWIATION OATHERED 

The following Is a conclensed, selective summary of information and 
inipresslons gleaned from our field visits. It is not intended to be a 
definitive or conclusive set of statements. 
1. Environment in which curriculum is used or may be used 
A. Target groups „ • 

Small cities—largest group studied-offer least anrount of red 

tape and hierarchy to deaVwith; key decision makers stand out. 

Sublirban areas— usually can afford program, often their schools 
already exposed to progressive or innovative materials ^nd 
pedagogies. 

Rural areas— often can't afford materials; often conservative; 
hard to follow up. • ^ 

Large metropolitan areas— most difficult to work with because of 

bureaucrcicy, difficulty in pinpointing key dacif^ion mnkorr.. 
Pol i tic', oi'l'-n (:r.'>r,'j i !:ic,!-.rrs lO bo. reluct nnt to 

a cu;i..«ii iK.Ciit i f- i.Iiti pruyi i-.^i;. 

H FniitlC'-'! chmatG runs the qa;nut fron. hard core couburvdlive lo 
liberal-progressive seeking all kinds of innovative curricula. 

C. Financial resources 

Sometimes materials provided through spec.ial innovative funding 
programs, or on an experimental, or pilot basis. 

Cost a problem to many small systems which simply find it prohib- 
itive. ' 

Same cost problem when an individual school wants the materials 
but is not supported by the system. 

• D. Educational climate {see remarks under Target groups) 

Some systems more interested in innovative curricula than others; 
interest cuts across urban/rural, rfassi fi cations . 

Often reflects political climate. 

Progressiveness of key decision makers influences educational 
climate. • 

II. Dissemination ' * ^ 

A. No .one process, across regions or within regions. < 

B Differs from system to system, but the more bureaucratic and larger 
the system, the harder the dissemiriation process, except where 
^ key decision makers seek out MACOS. 



C. Financial resources. of system or school again play a large role. 

a. Political climate an influence; the'more conservative the community 
the less receptive it may be to ftACOS. • 

III. Support 

A No oattern across reqions. Ranges from built-in support system 
^' ^l^rough su«rlnstitute training of social studies curriculum 
consultant for each system, to no support system at all. 

B. In most cases support is given on an "as can" basis. 

C. Problem for regional directors is the lack of funds and staff to , 

adequately follow up teachers. Regions quite large. 

D. Many teachers feel more follow-up would be helpful, even those who 

have taugiit the program for as many as four years New teachers 
to the program need beginning skills. Older teachers to the 
ppogram need revitalizing. 

IV. Functions 

thi id ran 

' ho^t peonte. trnm 1:r-ichr>rs tO' r-gion^l directors, sec ckilU and 
attitudab as the primary goals of the course for children. 
Teachers especially often stresi^ed attitudes toward self and 
interest in learning as important outcomes, 
• Both teachers and regional directors agree content is important 
• mainly as a mechanism for instilling attitudes or skills. 

Only proejram often cited as an existing social studies curric- 
ulum that could do the same thing for children that MACOS does.- 
■ ■ was Taba. Complaint about Taba was that it has method but no 

pre-packaged materials. 

Many teachers felt they could Impart same attitudes and skills 
through other materials ot courses. Others felt this ability 
was a result of exposure to MACOS training, Courses most often 
mentioned as possible vehicles were language arts » science, 
reading. 
Teachers 

Many administrators see MACOS as a good teacher training 
device, as do regional directors. See it as helping teachers 
develop child-centered teaching skills. In many instances 
teachers selected or volunteering for ^^^^"^"9 have been in 
principals' or administrators' opinion, among the better teachers 
in a school. 

Teachers felt r-IACOS had affected their teaching style; made 
Ihem more 0 en to children's inputs Often yo^^gor teacher 
felt they had been exposed to MACOS pedagogy in college methods 
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course and that their teachln- style was less affected than 
' did teachers who had been teaching for longer periods of time. 

Regional directors and administrators said MACOS spread of 
effect v/as evidenced by teachers* charlge of role In classroom 
(less teacher domination). Teachers expressed same sentlnient 
In different terms; "Am more open to students* questions"; "can 
tolerate 'good noise*"; "encourage debate rather than seek 'right' 
answers." - ' 

Often teachers .would suggest that they try to tie MACOS In v^ltli 
the total classroom, approaching other subjects with discovery 
and student-centered methods, or using MACOS topics as starting 
points. 

Some evidence of spread within a school ; non-MACOS teachers 
would notlde what was happening in a f^COS class, would 
express great 1r\terest in getting the course. Evidence of 
this Is scarce, however. 

B. Value of the course with different ethnic groups ^ 

Very little v/as- found on this point; an isolated few white teachers 
felt materials have different values for black children. Black 
teachers felt motoriaU. hnri iinivpr?iel valuor. . Asked if it could 

bo ponr. ible tr.:;1. viii:.v :!f n'i Iiimv Iki'.; whi t-'f lc;;f:hrrr ir.e liie i.vit'Mn* r ls 

'-oif! 'lie' "m.-.y \:.::vc lrt!iii',nj -..'l ih wiiitc leadiers , often had 
mid'ci'le class children to work with durina traininn inc: tifn^f^. . ^ 

C. Values of the- course across grade levels 

Most teachers use materi-^ls on 5th or 6th grade level and feel It 
is well suited to those levels. Those teachers using MACOS on 4th 
rrade level are split in their opinions. One group feels materials 
are too sophisti cated- for 4th graders. Others feel materials can be 
used with 4th graders successfully but not in as deep a fashion as 
with 5th and 6th. (One teacher said he v/ould like to use materials 
with children in 4th grade, then again in 6th. Felt there would be 
something new to gain at each level.) 

Some tea'chers using program on junior high school 1 eve T with, success. 
Feel it^.s a good introduction to anthropology. Also used as low as 
2nd grade vHth success. One problem- -where MACOS replaces traditional 
6th grade social studies some teachers feel children are penalized 
because they will be tested at the end of the year on traditional 
social studies content for junior high placement. In a few ce^es, 
teachars tried to teach both MACOS and the regularly prescribed 
curriculum. 
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0. Teacher Pre-service preparation 

Regional directors have conducted some pre-service training sessions. 
Either university classes or summer sessions, but evidence of their 
value Is scanty. No instances of pre-service teachers later becoming 
MACOS classroom teachers evaluated that we know of. 

E. Institutional change 

Mixed opinion on this from regional directors. Som feel institutional- 
change is a legitimate function of the course. Others feel no maaor 
change will cofne through only one course. 

Those who feel thalfinstitutlonal change is a legitimate function 
of the course see it affecting:^ 

School organization, e.g., team teaching 

evaluation procedures, e.g., doing away with report cards 

classroom norms, e.g., establish different norm which will ultimately 

establish other patterns^ of scHaol behavior 
tea'cher/principal roles,.e.g., some -principals have become more 
involved with students, teachers have developed new expertise 
and have been used as resources in other schools and systems. 

One regional director felt the pronram could become a vehicle for ^ 

unv.'flrcl mc'rilily for individuals, but that it moves thom up end out or 
11:0 s;;*. l.-;<! rci'n:r in.':;, conu i .^m Ii.'j ->y-.(/ji'i. 

Teachers provided some evidence and infnrmatinn in thp ,?rfl;is o^' 
institutional change. A few say they tiave abaiidoned A-F grading 
system. Several say that group planning or teaming has resulted 
from a need to cooperatively work out the schedules for films, etc. 
In some areas teachtirs are selecting other social studies curricula 
based on the adaptability to MACOS concepts and techniques. 
In goals section, many teachers mention wanting to have or to develop 
independent stu^y, non-teacher directed (new teacher role). 

In curriculum selection,' one administrator feels MACOS may be the 
answer to bilingual, bi -cultural problems; feels MACOS is culture free. 

V. Evaluation 

A Many teachers said they didn't use traditional "tests" to .evaluate 
' students' progress with MACOS. However, .rarely had they given up 
traditional grading for other coursers as a result of MACOS. Often 
teachers would express concern over the question of measuring 
children's progress in any objective fashion; most felt evaluation 
had to be subjective and based on the teacher's knowledge of her 
children. No teachers could suggest any test that would adequately 
measure effects or impact of MACOS. 

B Administrators most often took the position that teachers' opin^of^s 
subjective evaluations of the course' merit was their yardstick, 
along with observations of children, rep ts from specialists, parests, 
.and ot;hers, and their own judan^nt of the curriculum. 
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C. In only one Instance that we found was MACOS being compared in a 
systematic way with other innovative social studies curricula. - 

D. No evaluation Jieing done regionally with the exception of EME. 
Lack of funds and staff seemed to be the primary reasons. 

E. When asked about evaluation- for continued use of ^^ACOS, adminis- 
* trators again v/ere relying on teachers' opinions. 

In the next section, illustrative comments and more detailed information 
concerning certain issues are given. 
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FURTHER EXAMPLES AND TRENDS FROM INTERVIEWS 

What do te^cfiers perce1\fe as goals of MACOS? 

One of our Interests was what teachers and others see as the goals of 
the MACOS program. We would typically ask teachers what they saw as the 
goals for their students and received, as one might expect, a great variety 
of "ans\^fers. Soma teachers felt that the goals- are. primarily cognitive; 
learning to think, to make generalizations, draw inferences, analyze, make 
comparisons, etc. Others used these process ternis; learning how to formulate 
questions from data, how to verify', to collect and organize data,, to observe. 
Some referred to social skills, such as learning how to listen to others in 
groups, how to parti cipot.o and nxpress themsclvoG in groups, hov; to take 

V ■;:! ti ;^ ^i;^.. i u,u. . Su::.j tuvaJicPb 3 LrL^i^ied feeVinys and dttituclES as 
fiial cuncfrris . c=p?c1e!ly v.'itli their slow Uanters or poor readers. 

Exerpts from some of the interviews may help illustrate what we are 
talking about. A sixth grade teacher responded, "11/ goals are to try to get 
the students to look at themselves; build tdlerance for people different from 
themselves; to use inductive reasoning; to teach them to compare different 
things--man vs. animal, for example. The process is mor« important than the 
content." ' 

A teacher of fourth and fifth grades: "Concepts, What makes man 
human; the humanistic approach; have children deal with these higher order 
cognitive skills such as discussion, handling. data, generalizing, selection 
of pertinent data. [To view] content simply as a supportive concept." 

A sixth grade teacher: " [The course should help students to] think 
on their own without looking to teacher for direction' or reinforcement. 
Children in not-so-structurod classes are more used to group work than other 



children, with ability to take charge. Social relations is a super-goal." 

A group of sixth grade teachers: [The goals are to}"br1ng out the 
the uniqueness of man; appreciation of self, tlie human race; attitudes. 
Content is simply a means to an end. Students should think about self as 
as a human being that does have a place in society; there should be an 
appreciation of values and beliefs. Get them to think, to reason as they ^ 
have never done before; develop understanding of behavior, provide inter- 
actioja^o. they can observe changes in their own behavior." 

A teacher of fifth and sixth grades: "The children learn to 
make generalizations, draw inferences, develop ability to participate in 
groups learn to feel, comfortable with questions for which there are no 
right or v/ronn nnsv/o r5^." 

V..0 ludclios:. of iiun-grad.-u chlldruTi ayet] ten to tirirteen: "..Learning 
discussicn tcchniqua^* lo ui Ffereniiate between fact and opinion; the course 
arouses curiosity, induces follow-up to activities. They learn to deal 
with values; no right or wrong. Build up critical thinking, give children 
opportunity to think ; T^^n to defend opinions. [Learn about] human 
relations; everyone has differences, similarities. Teadies openness, respect 
for others." 

A teacher of fifth and sixth grades: "In reading and math we group 
by aliility; in 14AC0S we don't have to. Slow -learners have discovered 
they can think too. Most value for these children. No right or wrong answer; 
every child has ideas. ^4o5t important goal is ability to think on one's 
own; to learn to listen; to respect, opinions , not to say just anything. The 
course teaches children to think." 



«atv teachers coimented on the value that I«COS had for different 
cliildren. For example, they noted that the,^ was something for everyone, 
and everyone could contribute. They felt that the course was particularly 
good for "poor readers" or "slow learners" in a „un^,er of ways. Because 
Of the ™it1-„«d1a design of the course, children a,^ not dependent on 
reading as their mai^. or only source of information. Youngsters having 
difficulty with reading and writing could contribute by offering their 
thoughts and opinions. Hany teachers felt children who had thought of them- 
selves as unable to participate found that, they 'could have "their day in 
court." as one teacher put it. "Theyhave their chance to shine." 

What seemed very clear was that different teachers saw different 
values in. or had different goals, for the course. In so,«= cases, they 
boliov.-d thai: thv prio. ith... Uky h.vl for il,-:, ,-n. , v.. , ,'lr ■c^^. . 
ci.UJ wiui Ihe related needs ofHhe children t,h«/t:ught. ;.. „ther «se=. 
teac!:=rs did not fee', there was a direct relationship but that the goals 
served the needs of „,ost' children everywhere. Son,e teachers expressed the 
belief that the value of the course was that it enabled students to probe 
subjects about which they were relatively closed-n,1nded. For example, one 
teacher said she had found mcOS offered a way of opening up discussion of 
the racial and poverty issues In the community; that is. she had found the 
course served as a vehicle for talk about people from ethnic and socio-economi 
groups other- than those the' children belonged to. She feltlit was the only 
opportunity youngsters had to hear about and discuss people outside their 
own neighborhood and coiimunlty. 

The course was reported to be a vehicle for opening up discussion 
Of various value issues othervnse difficult to deal with. One teacher 
conin^nted. "We're always talking about education as dealing with values, but 
we never 'do' It." An elementary curriculum di.-ector said she had sat in 



Glasses where children had been discussing aggression and hostility In 

themselves « feelings she had not thought about until she was over twenty, 
us stie ptht- it* 

Mom are teachers evaluating the progress of students in the course? 

One of the greatest concerns teachers expressed was how to assess 
progress. Those v^e talked with felt that grades w^re inappropriate. In 
soma instances the teachers had refused to give grades. One group, under 
pressure to provide soma kind of feedback to parents, took it upon them- 
selves to have parent-teacher conferences, to describe what the children 
were doing an9 how progressipg. Some teachers have Instituted a system 
of written reports to parents; others have worked out a satisfactory- 
unsatisfactory form of evaluation. Still otlK^rs hnvo held to n^nifjn qrsdes 
at tiic ^>C.u:i\i/ ">.j3i:ilL:iicc; lliusc we La]l;uu io nut at all. happy about 
huv'Iiiy lu clivti grades. 

Teachers gave us a variety of statements describing what they used 
to assess progress. Some periodically used a brief vocabulary or knowledge 
test of their ov/n devising. In a few instances the teacher used the test 
developed at EDC. Some teachers have followed EDC's suggestion for inter- 
viewing children, or otherwise talking with them about their progress. ^ 
Many said they based their judgment on what they saw the child doing, 
an^ on .the extent to which the child participated. »The kinds of questions 

■if 

he raised, his role in group discussions, the enthusiasm and interest he 
evidenced were all considered. Many teachers we talked with felt that 
tests of any kind were inappropriate* or irrelevant. Som§ used self-made 
tests periodically as checks on progress; here a few noted that they used 

them mostly to have something to show to school or parents if need be. 



Evaluation poses a dilemma, for teachers. On the one hand, 
they know when children are interested, enthusiastic, and actively Involved, 

and for many teachers this involvement in itself seemed to be a major 
acconplishn^nt. On the other hand, the problem of how to coiwnuni cate 
progress disturbs some teachers. Furthermore, the way to improve their 
handling of the course or to improve the course for certain children was 
unclear. One teacher commented that training for the course should include , 
learning how to lead small group discussion. She saw her lack of training 
in such discussions a limiting factor when she first started MACOS. 

When pressed, teachers would describe cases in which they felt the 
course had not been successful with certain students. One estimated that 
of thirty ntufi-Mils around tenty-fivo v.'ould cvontuany bocom involved 
in •c!.c course. Soi.,':. di.£.cribed youiiostcrs who n.cv-r really, bscair.e 
1iilfcre:.lcu. We were told about a class which remained uninvolved so that 
the teacher finally abandoned the course. In short, we obcained some 
limited coroborating evidence of the ranges of preferences and reactions 
of children which were described at length by Hanley, Whitla, and, others 
in theix^evaluatlon of MACOS. 

What "do administrators, curriculum directors, and supervisors loqk for? 

As with teachers, there are different points of view and different 
expectations. It is probably not unfair to say that many administrators 
see the course as a vehicle for changing teachers'* attitudes and behavior 
.in the classroom. This'is not at all to suggest that they ar§ not 
concerned with values for children, too. When asked what they would look 
at in evaluating MACOS, many administrators and curriculum directors 
said they would be guided by reports from teachers, and by observations 
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they made of chlldreT. -^^dijewna for adni1nisj;rators and others is that 
they are dealing with a one-year cSxirs^.^ Problenis of what to do after 
f4AC0S. and how to restructure the whole ele^ntary social studies curric- 
uluift are of concern to various administrators. No one we talked with had 

I 

any formal plans for evaluation. 

It Is worth noting that, some supervisory and administrative per- 
sonnel felt specifically that one of the desirable features of the curric- 
.ulumwas' that materials are there. In their opinion, there are irany aids to' 
the tea,cher to help develop the pedagogical approach encouraged by the course; 
also, one does not have to spend m inordinate amount of time assembling 
additional materials. By the same token however, soms of the frequently 
rocurrinn proMiM::^^ n-ntionf^d -by .tnadicrs hnvc been IpcI: of filnn. film? not 
b.ir.Q av.U.u-!. v.'I..-.. n.ud.d.' lack ol a prujocior. -laiu delivery of booklets, 
ttc. 

Can anyone teach MACOS? \ 

Host people we talked to thought probab\y not, although the reasons 
given varied considerably. One center director, for example, felt that 
strictly reality-oriented teachers, to use his terms, probably could not and 
should not try to teach M/tCOS. By "reality-oriented" he meant having a need 
to deal mainly with facts and correct answers. The director also noted that 
strictly reality-oriented children probably should not take MACOS. 

As one way of approaching the question of who could or should teach, 
and who he should teach, we asked a number of people how the course might 
fail. The mo^t conanon answer was. "if it is taught as a traditional course 
by lecture, with emphasis on facts, anikwith tk teacher engaging in single 
response questions and answers." In connection with this question 



m hazard the guess, based on discussion with. center directors and otherSj, 
that so far the majority of teachers \^ho hav^ been teaching MACOS hav^ been 
a fairly select group, either handplcked fay the school system, or volunteers. 

We also hazard the guess that this question will become increasingly 
Important as the dissemination of the curriculum Increases. We found one 
case, for Instance, In which the board of education In a district adopted 
I4AC0S for all fifth grade classes next year. A principal In this district 
was puzzled by a small group of teachers who had made It' clear that they 
did not want to teach MACOS. One reason they gave was that they had seen 
last year's riACOS teacher spending'a consider'able amount of time borrov/ing 
and returning films, and othen-zisfi engaging in what seemed to be a good 
deal of extra and unnecessary 'work. We arp not tryinri to say that those 
i,:c;c:;.j.J^ v.'l i i lu/i. s;o\,c to Icacii il-ACUi u^Ai year;^ at this i^oiiit in 
i]mj Uiny do nor look ton/ard to it. Even in groups that liave lauylil MACOS. 
there are teachers who are not entirely enthusiastic about It. We were 
told of one who had not cared for the course at all, had been very dissat- 
isfied witit it, did'not like its orientation- and structure, and had felt 
very negatively about it by the end of the year. Wc do not know how the 
teacher's class felt about it. 

The general point is that behind the patent success and acceptance 
of MACOS there are some real uncertainties relevant to adopting and imple- 
menti-n'g it. There are few guidelines to fall back on in working through 
the, implicit decisions. It is not just a question of whether teachers are 
traditionalists or not; a number we talked- with openly and emphatically 
•declared themselves traditionalists previous to teaching fiACOS, and now 
are entirely differently oriented. Some consented bluntly that MACOS had 
shown them how poor their teaching approach had been. In observing classes 
of teachers whp describe themselves as previously very traditional,- it seemed 



clear to-some that whatever "traditional" meant before. It does not apply. 

now. Ft Is not just question of simplistic classifications; much 

depends,, it appeared to us, on the training that a teacher receives in the 

use of the course-, both befoi'^ and during presentation cf'it, as well as on 
the teacher's needs and oripntations. ^ 

Conte^it Considerations 

MACOS* is a one-year program, and as such raises a number of 
questions abouj^ hcM it fits into various school contexts, programmatically 
and institutionally. With many of the people v/e interviewed, we tried to 
.explore dimensions of the relationfhip betv^een f-IACOS and the contexts in . • 
which it is being used. We asked teachers, for example, "Do you think the 
abilities and tittitudes that children develop w1t.h MACOG vnll . continuo 
a:. I'l.: •^fJM.ch ...i'V. 'iiiLj I'r.c. iii'j!''-^ oir.alci'/:'" liic-ic v/uTC diri-crcnl rci-iioir^e 
ZzuZ UuJ.wi:* Tul i. pubsinnbtir, their general position bciiiy Llial it uU 
depends on what teacljers of hi cjher grades do. They thought that if upper 
grade teachers were tradit'fonal and approached social studies as a lecture 
course , then there woul d not be enough cany^over from I4AC0S. Some teachers 
hoped that the children would put pressure on subsequent -teachers 
continue with the kinds of instruction and learning that ha J been used with 
MACOS. Teachers of chil^Jren who took the course last year are reported by 
other teachers to be impressed by the children's immediate interest in 
discussion and free contribution of opinions and ideas. The MACOS children, 
according to these teachers, seemed more ready to react positively to new 
subject matter in social s-^udies.. - • 
• Most teachers, however, who have taught MACOS before did not knov/ 

vvhat had happened v/ith respect to children in subsequent years, although 
nearly everyone expressed great interest in knowing the' outcojue , and a need 



for feedback, tacfe of feedback was particularly noticeable wheri MACOS 
was taught at the sixth grade level , with children moving on to another 
school for seventh grade. In departmentalized teaching situatldns it is 
our Inpi-^sslon that there may. not be much cross-feed and feedback. 

There we^ teachers who felt that t4AC0S builds a good foundatibn and 
potentially develops lasting attitudes. These teachers believed there 
would be carry-over of effects almost regardless of what happened subse- 
quently. In one case, teachers and principal had i-forked on a follow-on 
program to MACOS to be .Impleinented in sixth gr^ade. The follow-on was built 
around further development of the five humanizing forces emphasized In 
MACOS. 

VJe also asked teachers "Are tliore other subjects or curricula In 
t.hlL <jc.!ko1 ut thi;: l.v:;! Uiuc uk,)' the saii.a or simi lar 

^r«n..ri<^Hn^^ rl'.illG, G-.- ^ti;uJL::> iit ui 1 lurc^?! «s ^V\'.U5^•^'" Hcrs again there 

r 

was a variety of answers that seemed to depend, among other things, on 
the teaching situation and the school organization. Generally, v/e found 
that where teachers saw some continuities they were with science, readin 
and English language arts, and occasionally mathematics, especially when 
manipulative materials or enactive approaches are used. Where classrooms, 
were self-contained, teachers v/ere very likely to say that they were using . 
more child centered approaches to teaching in various subject areas, 
applying where and as they could techniques they attributed to t4AC0S. In 
a few schools v/e visited where there were highly flexible arrangements such as 
team teaching, and multi- aged , multi-graded organization of children, 
teachers were likely to see most of the v/hole school program a$ consistent 
with and supportive 0# f^COS Instruction. Teach'ers in self-contained 
classrooms v/ere apt to regard themselves' as very different from the rest of 
the school staff and program ("this is a very traditional, conventional 



school"), and to i^gard little else as related to or consistent with 
HACOS- 

VHth respect to social studies faught either prior to or concurrent 
with r4AC0S. the typical response was that t^COS was- entirely different from 
the rest of the social studies program. Many teachers contrasted it 
strongly with what they regarded as traditional, uninispi red social- studies 
curricula! We e<icountererf a few cases in which other, new social studies 

i 

projects had been tried, such as- the Taba social* studies curriculum, and 
teachers seemed l^iclined to regard MACOS as similar but better in the 

V 

sense that contfent and material were available to support; methodology.- 

V^?e asked teachers whether they had found any aspects of the 
MACOS training or curriculum- had affected their teaching in other subjects, 
for ^i''p 'rU..'nt.ci1i.:c(J social studies teadfor:. th? Vi'cstion \.'rc not 
applicable. For others, there had been a spread of effect as described 
above, or else it was too early to tell. A problem for administrators 
and curriculum directors particularly arises over the necessities of 

' rescheduling social studies that would normally have been taught during 
fifth grade, for example. This was not just a matter of subject area re- 
adjustRffint. such as rescheduling American History. In some cases the 
question arose about where to pick up the teaching of such things as 
library skills, graphing skills, and the like that were viewed as integral 
components'of prior social studies programs. In this connection, one of 
the regional center directors noted that a key point one should rementer 
In going before a board of education with a curriculum is "to think through 
and be prepared to discuss the implications, and how it will affect the 
rest of the program." ., • , 



Views and experiences of the regional center directors 

we explored a nuinber of aspects of dissemination. distribOtion. 

use. support, and evaluation of MACOS with regional center directors. 
Vte won't try to describe in detajl all of what was learned, but certain 

points are worth noting here. . ■ • 

One question we were interested in was what standards of usage 
or procedure had been required or encouraged by regional center directors. 
Ther* are various aspects of this question, and responses varied somewhat , 
from center to center. Here is a summary: . 

"EOC requires f1 ve . teachers for each film, but the practicality 
of t'-e requirement depends on distribution and size of classes, and 
commitment of the central ndmi nitration. There 1s a need for the curric- 

;„ UUW.n.LluH, u"aM,;.,y. o„u Mua M ly cuul.ol. Ti.s curriculum 
coordinator's -authority is needed, and should be established ahead of time." 

■ "The amount of instruction time required is forty-five minutes ^, 
a day each week, or one hour four days a week. Haven't had oth^r design 
standards; the course itself demands certain arrangements." 

"hight be desirable to require or Encourage standards of procedure.- 
but whether that is practical depends on bureaucracy involved in the county, 
and the number of teacheVs who want to try MACOS. Unless the regional 
director works with a school system where. he can be selective-, he takes 
what he can get." » 

"We encourage a commitment from the district to use the MACOS 
materials to implement the program. We encourage a team, with at least 
one administrator, to con^ from each school district to the su.™ner institute. 
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The team should be strong teacheri^ with good leadership potentia? and 
pood relationship with their peers. 

-I feel it's the job of the regional director to visit the school 
principals and administrators to help create the support necessary." " 

«We demand attendance of every teacher at a workshop If she's 
teaching the course for the first tin^. Workshop models and format vary. 
We encourage non-MACOS teachers to attend workshops for credit. There 
should be entrance criteria for teachers to get into MACOS, but why^ 
e>5clude resistant, traditional teachers?" ^ ^ , 

t 

Our observations within and anionci regions did not lead to any 
. ..... ,. ... .... „; ] i:ic-;r.;iiPi bu.v.:'Ch c'icfc-ii.ii.nlion or tri^inincj 

- ' - ..ou.c T;...c wc. ■InJiuaLlun., huvev.r, inal Selection. Lvdiniuy, 
and utilization processes and standards are important, can have effects, 
and could be further investigated. 

Our general impression, based on tb.is brief study, is that the MACOS 
curriculum is an extraordinary achievement on a number of counts, but that 
knowledge about its use is presently characterized by a large degree of 
indeterminacy or uncertainty and only partial explication of its educational 

pYoperties and effects. • 

Overall. MACOS appears to have tapped several major trends "or thrusts 
taking place in the social studies domain in particular, and in educational 
philosophy in ssneral. Within the social studies program at the .elementary 
, level . there Is a growing thrust toward seeing social studies as a vehicle 
. ' for teaching Inquiry skills pertinent to later effective citizenship. The 

approach described by Masslalas and Cox would appear to be an example of ^ 

f 
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this thnTst."^ There is also a strong trend toward development of 
enrational and i magi national growth and coifipetehce in the service of educa- 
tional objectives, a position exemplified by Richard Jones.^ Underlying 
other interests in social studies is the concern with the continued Impart- 
ing of subject area knowledge in a traditional sense. Yet another .set of 
trends seems to conceni a relatit)nship between social studies and other 
' ar^as. such as science. Here there is the position taken by Gagne'3 that 
process skills^ applicable in physical science are readily generalizable to 
other areas of science, including presumably the social sciences; however, 
' Jones proposes the hypothesis ttiat there are essential differences in the 
focus of the physical sciences and the social sciences from the point of 
View of the learner, with consequent differences In the nature of success- 
and failure in the two areas. . • . 
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Massialas, Byron G. and Cox. Benjamin C. IjBijlDL. ^" Social Stu dir.s. 
New York: McSraw. HilT. 1966. ^ 

'2 • oones. Richard M. L^IltiiOalJl^li"^ ^" Education. New York: New York 
University Press, 1968. . 

3 rW Robert M Psychological Issues in Sci ence-A Pj-oc gsij^roach.. 
in Jrfcin Association for tho'Advancoinont of Scioi^^iTWHychologi-^ 
Bases of Science-A Process Approach . 1965. p. 1-8. 
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" . 'some critical features of the M^SQ^CURRICULUM . 

These appear to be sons of the crltjfcaT features of MACOS that 
1 were expressed directly or Indirectly by varTous people in the field. 

1. The absence of a textbook as the core material 

2. The central role of filins 

3. The scope, depth, quality, and applicability of available 
• ; materials . ' 

4. The pedagogical and educational orientations of the course 

5. The training component of past dissemination activities 

6. The conceptual structure of the curriculum-, multiple disci- 
plines, spiral curriculum design 

/. The intiHi-«ie.<itd design ^ 
C. Th^ autOftOfflo«s, s«lf'cofltained |kick«3e oharac^ristie- 

9. The implicit and explicit value orientations 
ia. The lack of explicit outcome criteria and evaluation 
uncertainties 

n. Lack of right-wrong answers; teachers no longer experts 

12. Appeal to adult frame of reference and intellectual curiosity 

13. Group orientation 

14. Lack of closure-divergence-and the problem of subsequent ^ 
continuity 

15. Focus o1. humanity, inter-cultural relationships, humanness 
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CONSIDERATIONS CONCERfUNG FURTHER 
EVALUATION OF -MACOS 

After considerable thought, examination of a nuirf)er of aspects 
of the present state of affairs with mm: and brief exploration of trends 
ind issues that seem to be taking shape in social studies education In i 
the elerr^ntary grades, we offer the following classification of possible 
courses of action for the National Science Foundation to consider. 

1. What a^e the performaace characteristics of MACOS? (i.e. . 
how effective is it? what are its various outcomes?) 

a. with students 

b. with teachers 

c. v.ilh sclioolb. • 

Z. iJhat. ar. thP pprinrm^nro rPlat.l0n.hin5 of HftCnS to ot+Pr 

curricula? 

' ' • a. social studies 
-b. science, 
c. other 

3. \jhat are relationships of dissemination, training, and support 

, systems to MACOS performance? 

4. What are the relationships of mCOS to institutional change? 

5. What are the implications of mm for other curriculum 

development projects or approaches? 

6. What are. the relationships of different evaluation strategies 

to MACOS performance? 

Each of these questions will be discussed separately. 



1. Perfomance characteristics of HACOS (h«» effective is It?) 

Ther^ are seven major goals stated for MACOS.'' Of these, six 
refer to students as the target group, one refers to teachers. The goals 
are unspecified in the sense that there are no criteria provided for 
deciding whether goals are accomplished, approached, or missed entirely. 
Much is left to the teacher to judge. The Stratc a ie s for Evaluation , now 
available should alleviate son« of the uncertainties and perplexities so 
frequently expressed by teachers. The manual contains the short content 
Cuestionnaires (tests) that were used in the formative evaluation studies of 
Hanley. et. ai- . as well as classroom checklists, interview guidelines, 
guidelines for use of creative formats in assessing progress, and checklists ^ 
for .1 ,■...,.•„..„■, n..,..,va,-lon.. TlUTO .ro /hovover., no dcclam.d priori.tic. 
of goals, alUKHKjh it, soems clear enou.jh Lhal goals of methods and 
attitudes are, generally, of paramount priority. 

V see.s to us that the crucial characteristic of !«COS from the 
point of v'iew of performance evaluation is its. duration of one year. It 
,s a package that assumes no prior preparation of pupils and does not neces- 
sarily directly connect with subsequent curriculum subjects or units. The 
skills and orientations that children are encouraged to develop in MACOS 
are presumably relatable to prior and subsequent slcills and orientations. 
V,e say presumably since It is by n^ZnTclear to .s that the procedures and 
critena .re specifiable for de^f^ing, evoking, and Identifying inq.u1ry 
skills in so.e ordinal fashion; or in some gencrali^able fashion fro™ one 
subject area to another; or from one t1^ period to another. As Cole and 



1 Educati on Devel opment Center M= .AJojirsi^iitud^^ 
Strategies . Cambridge. Mass. . 1970. p. ii- 
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Seferian put It, "one man's inference is another man's classification." 

There is a need for careful pre-course observation and measure- 
ment, as well as for follow-up well beyond the end of the course. The 
pre-course observation is more than a matter of pre-testing for knoi^lcdge, 
intelligence, reading achievement, etc. It should include establishing 
performance trends in social learning behavior, inquiry skills, and attitudes 
or intei-ests of'children priorJ;o their taking MACOS. The follow-MP assumes 
especial importance under the condition in which children transfer to another 
school upon;;eoiTipletfon of the course, as happens frequently when it is given 
in the sixth grade. The presumptions are 1) that the course may have differ- 
ent performance characteristics with respect to any or all of its avowed 
goals v.'it!i children of difforf.'nt, previous habits and expectations and 2) 

that the course wi ll hcive dUfer^ntial- uxtcnded cansoquences according to the 

4 

different fates of chi ldren beyond the g^^ade in which it is given. 

The time-span of the course (and its implication^ is. in our opinion 

the single invariant property of MACOS bearing on continuing evaluation. 5 



^ Cole. Henry P. and Seferian, Albert. Analysis of process curricula. In Andreas, 
Rnrton L.et.al.. Research in Process Curricula . Eastern Regional Institute for 
Education, Syracuse, New york. Harch, 1970 . 

3- The Hanley report consistently documented differences in Pe^^?^'Jfnf^. 
tween inner city and suburban children. It called attention to different en- 
trrcharaaeristics of children in these categories. These are gross categor- 
ies. No analysis of variations within categories was made. 
^ In each case, extended pre- and post-course observation would bear not only 
on questions of differential performance characteristics, but, also on questions 
of 2hat schools might, do before and after the course to maximize its usefulness. 
^ There are. to be sure, other invariant characteristics of the package. A 
conspicuous one is the absence of a textbook and the centraHty f .^l^^^f • T'^^ 
av^ilabllitv of the films and other materials to teachers as needed may have 
.^onslquences ?or morale and performance. Another invariant, as already noted, 
is the v^j:: .ice of specific objectives and goals. 
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There are other characteristics relating to continuing evaluation 
that are not immutable or invariant. Ond, of central linportanco, is the . 
unfamiliarity of many teachers with the content of the course, and to a 
lesser extent, with the pedagogical requirements and i^les. The evidence 
to date is that students are having difficulty mastering concepts, and that 
the methods of Inquiry are neither extending beyond the boundaries of the 
course nor enduring. This may be in part a consequence of developmental 
limitations of children. Until it can be shown, ho\<rever. that teachers 
have mastered the course conceptually as well as pedagogically . the newness 
of the structure of the discipline to teachers has to be considered an 
important characteristic in continuing evaluation.^ The Implication here 
is thi't the c\m.-;th quail 1.1 of teachers ncoci to be exaniinod. Tins is not 
as straicjhcforward as laicjht appear, DeparLmcntal.ized teachers in fact may 
teach each unit four to five times as frequently as self-contained classroom 
teachers during the same calendar period. Most of the departmentalized 
teachers we talked with felt that each succeeding class during the day was 
jj^^fgy.gnt £er is. and for the teacher. One or two said that they sonietimes 
felt like a tape recorder at the end of the day. Teachers in team teaching 
situations of one form or another may be more like departmentalized or more 
like self-contained teachers., depending on the arrangement. The potential 
difference with team teachers, however, in speed of mastery of the course, 
is the opportunity to engage in more continuous examination of its thrusts 
and concepts through exchange of observations, ideas, information, and 
opinions. 



^ As Goodlad has noted, elementary' school teachers are typically.tra ned as 
«.neralists Goodlad. John. I. Direction and redirection for curriculum 
SSe. In^ee^r.^^^^^^^ ed. Curriculum Change: Directio ns and Process, 
rtiCD, NEA, 1966. pp. 1-14. 
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Another variable characteristic of the course that has, tgi our 
opinlcm. Important Implications for ccmtlnuing evaluattba of performance 
Is Its pedagogical and ccm tent linkage with other. segments of the school 
dc^y from the point of vieiv of students. It is not unreasonable to 
hypothesize that educational situations that reinforce behaviors, attitudes, 
and skills reinforced in MACOS should enhance at least the generallzable 
behavioral and cognitive outcomes of MACOS. This continuity or discontin- 
uity of Opportunity, utilization, and reinforcement patterns is not 
necessarily correlated with the organization of instruction (i.e.> self- 
contained; departnentalized; team teaching, etc.). 

A third variable that seems especially important is the large 
dcigrce of intarncil flexibnity of teachi-no an^anfjements and processes 
inhorcnl in t!ie courSu. The cour.su itself is highly structui-ed with respect 
to the seque.icifKj of units. The options for proceeding, the pacing, the * 
formation of groups of students, the opportunities for exploiting topics 
and issues of interest that may arise, and so on, all are sources of 
variability that potentially bear on performance of the course. It is this 
flexibility obviously that makes it difficult to think of MACOS as a treat- 
ment. This characteristic, in our opinion, poses methodological probl ems 
for evaluation compared to which most others seem pale. 

A final variable is the focus on the group as the operating unit 
of the couj-se. This is the design consequence of Bruner's concept of the 
function of the reciprocally operating group in learning. An implication 
of this focus is that, while the individual student Is the carrier of the 
'results of education, the group may be the more appropriate unit for 
observing and evaluating some of the results of the course. 

i 06 ■ 



• : The preceding points suggest that further evaluation (of 
the extent to which the course achieves Its. goals, given that they can 
be operationally defined), should take Into account the pHor context . 
of students'.educatlen. the concurrent context, the subsequent contexts, 
the^ teachers' nast^ry of the course, and the Internal organization and 

' conduct of the course, as well as the developmental level and abilities of 
students. Further evaluation should also use a variety of n^asures andl 

observation procedures. 

If it were possible to show reliable and operationally meaningful 
differences in perforaance' (effectiveness) with respect to any or all of 
the goals in relation to various operational conditions, on what decisions 
would such infomntion bear? Fssentiall.v, conWnuInn eval.>ati;on would 

L -d-ci.-,:... i.i local .U.col icv.l.. by r-o.-i .l-,.9 l-^'^ll'- ir,fur.lli.i, 
about performance .ndor various conditions of .input and Implcrxntation. 
It would also relate to so« decisions about further curriculum desigrt and 
developiTient. 
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2. What are .the relationships of MACOS with otl[ier curricula In social 
studies, and in other subjects or disciplines? . 

In the prececKng discussion, the MACOS package was treated as a 
device as^uch'. The questions arising from this concept involve what the " 
device does (or enables people to do), under varyfu^conditi^hs, and 
under vailing time periods. A second point of view is a consideration 
Of MACOS .as a member of a class, as an exemplar of a set. As such, it 
can be examined for its class implications, and it can be examined -coir^arar 
tivelyMn relation to other menders of its class. 

Viewed from this perspective, MACOS has certain characteristics 
that seem especially important for further evaluati-on. One Is the focus on 
mechcirilsmr. for survival, or more broadly, on structurrs and functions vis 
a vis tii-/i.rL!if... hi' . A sc-cunu, wliicii is a curoTlary of tha first, is tlie 
To^-ub uulii un social and individual Lchiivior- diid their relationships to 
motives and feelings. Im othe^ of particular significance are the 
multi -media design (and the realistic, non-expqsi tory nature of the filns) 
and the extensive array of materie^l in support of content and teaching 
process (Including lesson planning and preparation). A iMnal characteristic 
for comparative evaluation is the spiral design of the curriculum. 

The implication of the focus on mechanisms for survival ; 
is that the course has a basic value orientation, or at least a criterion 
for making judgnents; namely, survival. In this respect it is not simply 
descriptive. Therefore, if it were to be compared with other social studies 
courses, for example on whatever measures , it would be appropriate to 
consider specifically the role of the underlying value orientation (or the 



7 

The various classes of which MACOS is a member include process curricula; 
Inter-disciplinary curricula; student-centered curricula; mul ti -media curricu- 
la; social studies curricula, etc. 

o . ■ 
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presence or absence of one). • . ■ . 

The Implications of. the focus on behavior are for the jrotl vatiiig 
efficacy of the course* as well as for educational outcomas. The multl- 
iQsdIa design, diverse learning materials and spiral design have In^lica- 
tions for measures of efficiency as well as effectiveness. It -is possible 
that, other things equal, the muUI -media design has the effect of broadening 
the range of students who achieve particular levels of vocabulaiy." concep- 
tual mastery, interest, etc. Similarly, the availability of materials 
may in fact have effects'not only on learning time for teachers, but also 
on general acceptance by teachers. vj 

These considerations' suggest possible directTions or forms of 
■ coninnrr.tiv.^ evaluatiojjt . In rddU.ion to comp'ifi ^:nns of ac^rformancc in 
cocjnifiva, /.ffL^ctiv^, and bcha\riorul domainb with students, and in ssmilar 
domains fol teachers as teachers, it wou^ld also be appropriate to compare 
MACOS witJ other class exemplirs in-tenns of scope of effectiveness. insialV 
ation effjciency, and implementatibn efficiency. Comparative evaluations 
would supLrt or relate to adoption, deployment, and support decisions a tV 
local anJ state ■ levels. They wou^d have greater bearing in the long run, 
hw/ever, on curriculum design -and development decisions, we believe, and 
possibly' on decisions concerning teacher preparation and in-service training 



3.* What Is the relationship of MACOS' perfomiance to disseoii nation, 

training, and support? ' '' ^ / 

The cluster of Regional Cqi^ters established for ^^COS has been 
extremely Important In the success of the curriculum tc date. While tftere 
has been substantial variation in approach'among regions (within the general 
limits of the model set up by EDC), there is ample evidence of the values 
of the roles played by the Regional Centers, and of the implications of 
variations in roles. Yet there are few data readily available bearing on 
evaluation of roles and methods. Only one center has systematically 
•collected and analyzed information about tlie use of the course by teachers, 
within its region. There has been wide variation in record-keeping among 
centnrf;* and no rcqui rcm?nt and funding for bnr,1c data, rocordi ncj related 
to parui..:ilers of dissciiriiiution , training, and support. 

Do dissemination, training, and support affect performance? 
Are there optimum approaches? In what ways do, they depend on characteris- 

tics of the receiving and sustaining environments? In wt^at ways do tliey 

* 

depend on the curriculum or materials? ' , 

Obviously, dissemination affects performance in a trivial sense: 
no dissemination, no performance. Beyond that, there are a host of questions 
about processes of dissemination, adoption,, and support that can eventually 
be related to performance. 8 The need to train teachers to use^^the course 
seems self-evident. But hpw effective are different training 'models and 
methods, with whom? There is evidence that some teachers who go to the 
MACOS summer institutes to become teachers of teachers during the school 
year ma.v not, in fact, follow through vfith the expected seminars. Is 
that because there is inadequate incentive? Inadequate planning and prepara- 

® ' See , for example, Carlson, Hi chard 0. Ad option of Educational Innoy atjoni. 
Center for the "Advanced Study of Educational Admim strati on, University of 
Oregon, Eugene, Oregon, 1965. 4n 
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tion? What Is the consequence for the performance of the curriculum with 

■ 

teachers as well as with children? What are the consequences for perfbrm- 
ance of different adoption processes or modes? So far the course' hts been 
very successful largely with teachers who have volunteered to learn and 

teach It. . 1 , 

What constitutes support? One teacher with whom we^lked specifi- 
cally recommended that the regional center should be much more'actlve in 
supplying Information and literature about mcOS utilization, techniques* 
and developmants. The MACOS newsletter establishtH by ERIE for Its regional 
center is one. excel lent form of such support. In another case, we. talked 
with a district director of social studies who was working with teachers 
using HACO^ on- tho prohl f-^nr. of evnlustlnn di scir^si ons by students ovcir 

rncU is a fona uf buppiirt lor tiie course.. One principal expressed 
concern ove. liie Idck of Publicity and feedback concerning tiae use of the 
course nationally. Is a form of support needed here and would it affect 
performance? Ttiese examples are intended to convey a range of possibilities 
beyond the obvious logistics and supply forms of support.. 

Some of the characteristics of MACOS that may be especially related, 
or possibly sensitive, to dissemination, training and support strategies and 

methods include: . . . , . -xu -i ^ 

a. the clash of the course with established curricula and with value 

systems 

b. the interplay betv/een the highly organized structure of the 
course and the reinforcement or encouragement of divergence 
among children in its iinplementati on 

c. the emphasis on and opportunities f Disengagement of emotion 
and cognition in children toward educationaT goals 

d. the shifting or multiple requirements of the roles of the teacher 



' e.^ the unpredictable Interpl^ of neecfe. and expectations Qf dilldren 
$*nh the orientations and requirements of the course 
Mould evaluation of dissemination, training, and suj^ort systens 
4)r model^ be feas1b1e*^at this time? Our opinion Is that It wA^d be, 
^perhaps wiVh special a r ran patients With the publisher. Curriculum tevdiopment 
Assbclatfii*^ ir^h^^ focus Is On generallzable variables and models, the 
-value of ^Mch smidy could be great In the long run. 

Evaluation ^f various adoption, training, and sypport processes 
anrf models would relke directly to adoption and Implementation decisions. 
It wpuld also relate to decisions Sbout the requirements of overall 
developnfent. distribution, and support systems In future curriculum produc- 

♦ 

.tion efforts. 

4. Relationships of MACUS to institutional change 

The curriculum can and does have affects on schools and comnunities. 
In the casei)f flACOS these have covered'a range of react16r)S. from highly 
positive ("for the first time my youngster is talking at home about what 
he's doing in school") to highly negative (concerns about evolution, sex 
education, etc. . occasionally emerge). Similarly, reactions of teachers, 
students., curriculum specialists and others have varied, as one might expect. 

There is little evidence that we could detect of institutional 
change related to MACOS. There was evidence of changes within school 
buildings. The course has led some teachers to challenge the grading system, 
at least as applied to MACOS. Other examples of changes stimulated by the 
course are further curricular changes for earlier and later grades; changes 
'in school-home or teacher-parent relations; changes in supply distribution 
activities; changes in relationships among teachers; changes in views of 
teachers, administrators, parents, etc., of what an educational course is 
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(where Is the text^ok? -^^y don't you have homevmrk in a book?); and so 
on. There are also potentially significant changes Iri the roles of the 
teacher and the student, indeed, this Is one of the goals of the course. 
But with the possible exception of this .last point, these are «K)dest changa&y 
ci^pparedwit h the swee ping institutional changes and r efof nis so urgently 



and Increasingly called for. For instance, the course may change the role 
of the teacher in the claasroora. but it will not gorse affect his status 
" and role InHhe system. It will not affect the organization of educational 
activities, though it conceivably could contribute to redefinitions of 
educational goals. Nevertheless, m a local level some of the impacts of 
the curriculum may have ripple effects or synergistic effects over time. 

One group at a regional center said it is-expecting too much of 
one lUnc^ r.ocir.l s^ludicr. curruulu;n to H'J: mt It lir.vc- Much insM 1 u' i om.I 
imoact. ThPy are al.nn.t c^H^^nly right. It is cf intcrcGt, hov.'cvcr. to 
individual schools, teachers, parents, and others to know more about the 
possible secondary and tertiary effects of curricula. Evaluation focussing 
on such impacts would provide information relating to decisions concerning 
strategies of dissemination, adoption and support locally, and at higher 
levels (state, regional, and national). 

5. Implications of MACOS for other curriculum development projects 

The Man: A Course of Study curriculum can be examined from yet 
another perspective. One may explore the implications of the MACOS 
project and its outcome for further curriculum design and development 
efforts. An evaluation of the curriculum from that point of view would 
encompass a much broader set of data than even a comprehensive appraisal 
of the course ger se. 
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One knowledgeable director of instruction in a school district 
we visited asked how realistic it is to expect teachers to develop courses 
during the summer, considering the time, talents, and resources that went 
~" into MACOS. Does this country need a curriculum design- and development 
m achin^'y ^» tiiP srale. ssy^f l_sfi_ace ex ploration program? 

Goodlad has called attention to the direction that curriculum 
■ reform should take.^ What kinds of funds and organizations are needed 
to fnov6 in the directions Goodlad has suggested? Vldt^fd an analysis of the 
MACOS project and curriculum yield Information leading to important decisions 
^ about the investment of resources and the creation of new alliances and ^ 
inter-organizational arrangements? On a more mundane l&vel . there are 
■ • implications for curricuUnii design in the conjunction of content and 
' teacirincj rules .nd ...tl.oas. THlmo arc, obviously, inipVi cations in the 
multi-media fonii of tho course, in the inclusion of games, in the forms 
and sequences of contrasts, in the completeness of materials specifically 
designed and developed around the conceptual structure of the course. 

Evaluation of the project and the "curriculum oriented tavard 
this issue would bear to some extent on decisions iiwolved in subsequent 
design. It would bear most directly on decisions about the allocation and 
organization of resources for curriculum change in the social studies and 
possibly other areas. 
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6. What are the relationships of alternative evaluation strategies to t4AC0S 

performance? ' 

The purpose of the present study was to suggest alternatives for 
the National Science Foundation to consider with respect to furtiier evalua- 
tion of MACOS. Tt islfppn)prTate^^ 

subject of evaluation strategies as such. Let us shift the focus from the 
question of what strategies are appropriate or feasible to the question of 
the properties of evaluation models and approaches £er se. Let us consider 
evaluation models as a subject for research and demonstration, with MACOS 
as the occasion or vehicle for the examination of dlyfferent models. 

There are several evaluation models currently receiving wide 
atlcntinn. VJlrilG all rvnludtion pro^^unHihly hrr. to do vnlh decislon-ir.r-l'.ing, 
different modols seem cippli cable to different levels of organization, ; 
to have different operational implications, and to have different utilization 
characteristics. Stake's transactional model, for example, appears to be 
aimed at the immediate operational level -the class room JO There are strong 
parallels between Stake's conception of evaluation and vlones' descriptions 
of experimental lessons in the development of MACOS. "^^ Stufflebeam's 

TD TT^ Robert E. The Countenance of Educational Evaluation. 
Teachfrs coll eoe Record 68, April. 1967, 523-540. For two applications of 
Mp'roeneran m-'^see Chapter 5 by Koland F. Payette and Benjamin C. 
SffNL^Dinrnsion^^^ Evaluation of Social Studies Programs), and Chapter 7 
irviL Slett W W Stevens, Jr., and Celeste P. Woodley (A Mods for 
5^a v^inl cS^ricuf^ and Classroom Transactions) in Dorothy McClure 

fra er . Id. ? S^daT^^ Development; Prospe cts and Problen^, 

. 39tr!rearboik7-TiOT?ii^^ 

Oones, Richard M. , op. cit. 



CIPP im)de1^2 focuses on a broader operational sequence and seems to lead 
to a wider range of deci si on -making levels. Provus' inodel^3 appears most 
applicable to decision-making at the level of the school district. This is 
not intended to be a comprehensive survey of evaluation models. The 
pcint 1^ tHat th^re are a number of alternative modeljjHthjmry^ focuses 
and varying pt^sumed operating characteristics. The ^iACOS curriculum, we 
suggest, could provide a vehicle for exercizing or examining severaf 
alternative evaluation models for the purpose of clarifying their utility 
and operating characteristics. ; 

The curriculum is no longer in a developmental stage. Funding 
decisions concerning research and development of that particular curriculum 
are ovnr. But the impleincntati oru r.aintcnance . and improvon,c?nt dncision- 
situations hav. really just begun, Kurthermore r.ACOS is sufficiently 
large and costly to have extensive educational and administrative consequences. 



T2 c+.,ffiphpnm Daniel L Evaluation as Enlightenment for Decision- 

Abetting Insightful Action. Columbus. Ohio: Evaluation center, me un u 
University, 1968. 

13 Prnv.i<; Malcom The Evaluation of Ongoing Programs in the Public School 
Systemy'y^T^rRalJh . ed. • |^-^?|y{flH^ 

l^ans. National Society for the Study of Education. Chicago, university oi 

Chicago Press, 1969. 

14 . e a see Evaluation For Administrative Action. Journal 
Sl?,?^."frH„r°ttona ) Research! Vol. XXXVI 11. No. 1. February. 1968. 
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Yet it is sufficiently delimited to bQ readily observable. It is presently | 
being used in a wide range of envi ronnrents ; there will in all likelihood 
, be greatly increased adoption of it in the next few years. ' . . .'""I 

The question posed here is-, in what ways do various evaluation 



models or strategies affect r4AC0S perfom\ance? While an appropriate measure 
of effectiveness of evaluation processes Is their influence on decisions 
■{at whatever level), these decisions will relate to the utilization and 
effectiveness of the curriculum in one respect or another. Some evaluation 
models, especially those that are avowedly system analytically oriented, base 
their claims for utility in part on the power: of feedback loops". An important 
question about such models 1s, what are the operating characteristics of such 
fCL-Jln^ch locpsY rL'jdl.uilk alouiuls. There is need to expl icate planned or 
p)c:>uhitiu ftitidudck loops to Improve the usefulness of educational evaluation. 
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AITERTIATIVE STRATEGIES OR APPROACHES 

lo this final section we shall discuss specific approadies to 
further evaluation of MACOS. We shall offer at least some analysis of 
-the-^dvantages and limitations of each. 

In the preceding section we discussed briefly six evaluation 
Issues or categories of questions about f4AC0S. Here we shall be concerned 
with three major alternative approaches to further evaluation: 

1. absolute evaluation'-eyaluation of the performance characteris- 

tics of the curriculum. 

2. comparative evaluation of the curriculum 

3. svstenrlc evo1 u.nlion— cval u^^tion of the systems affected by 

or related to the curriculum {this incorporates points 3-6 in the prer^HInn 
section). 

These are not mutually exclusive choices, necessarily. Comparative 
evaluation subsumes absolute evaluation (or at least aspects 
Qf absolute evaluation). Either absolute or comparative approaches could 
Include systemic evaluation. We think there are advantages, however, 
in considering them separately. 
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/Absolute Evaluation of MACOS 

- MACOS was not designed according to a planned structure of specific 
behavioral objectives. It' seems characteristic of curricula designed to 
facilitate open-ended teaching rather than goal-directed teaching."" It has 
seven goals that can be operational i zed. There are 65 units, each capable 
of being evaluated. There are test items th^ have been used in the forma- . 
tive evaluation of the curriculum, and which can be used.to n^asure certain 
goal achievements. The items are few in nunter, and it is not at all clear 
how they sample the universes to which they belong. They are highly specific 
to the content of the course. Norms are available for the Animals section, 
but not for the Netsilik section. The questions for that section are 
intended to bo di.-(jno3t1c toOlV for the tuachcT, not re>1ikincj flavices. 
The cvaTui^ro, during HACOS' developivciit, ihuUc eAleiibive Ube of h.le.vlevr.iiy 
as a data gathering technique. They have provided teachers with construe- - 

* 

tive guidelines for using the same methods themselves. 

" Further appraisal of thu curriculunj., using as a minimum the methods 
and instruments employed in the formative evaluations , is obviously 
feasible, depending on the criteria of reliability and precision one sets. 
Our investigations have led us to suspect that further study designed to 
'replicat? Hanley's work will eventually show, perhaps more elaborately 
and in more readily interpretable form, that Hanley's results, both pro 



1 cf Lee S. Shulman and Evan R. Keislar. eds. Le^^ing by Discovery 
Chicago': Rand McNally, 1966. See especially the edT^FT^^ 
comments about curriculum development, pgs. 185-190^ 



and con, a,^ substantially pliable with aspect to outcomes for students 
and S01B aspects of teachins style or technique and classroom managenent. 

(tore Instruments and items could be developed to yield more 

explicit and -hianipuiatable neasures of HACOS goal -related behaviors. 
-mB^-conWTmth>rob**1ysho«M-4^^ 

taken by groups or classes of childrert. as well as measures of individuals, 
categories of possible outcomes of MACOS. even In a non-conparative | 
evaluation, could and no doubt should be extended insofar as observatioh , 
,1s concerned. Cronbach's suggested list serves as a useful starting p^int 
for such an extension-^ It is not clear, however, that development of an 
evaluation mouel based on arrays and hierarchies of specific behavioral 
go.ls .nd cjo.l crit^ri,-. v,ouUi ba an undortokinn vJorth doing for IIACOS. The 
'elegant moctel "developed for Sciegce^A Process Approach^ applied to a 
curricul>.n with a theoretical base and content quite different from HACOS. 
There of course must be criteria for categorizing observations, and, in 
some cases, ordering degree or quality of complex behaviors such as hypothe- 
sizing, generalizing, listening to others; 'and so on. There are instruments 



2 , Cronbach. Lee J. . The logic of experiments on discovery. Ch. 5 In 
Shulman and Keislar, op. cit. 

3 AAAS Con^lssion on Science Education. .Msncer^o^i^ 



available that enable observers to rate the occurrence of particular fonns 
of betiavlor. or the operations presumed to intervene, between stimulus and 



beha/ior. Criteria for classifying and rating responses'can b.e' applied to 
Interviews and to the interpretation of paper and pencil test items. 
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Measure's of perTorir^ali^e Tor"mW wi ~ 
hierarchies of impact value in the following sense; 

1. Acquisition: rate of occurrence and/or quality of particular 
\ dimensions or kinds of knowledge, skills, or attitudes 

\ (question posing, outa gathering, categor1zing,atti tudes . 

orientations, concepts^, models, etc.) 

2. Lateral transfer or extension: amount and kind of application 

. of concepts, ideas, orientations, methods, behavior, attitudes 
' In conccrnn, si tiMtions , -or topi^.^ other tlinn tho^.r. of th'- 
Cgur;.L. uuriiuj Liiilu's invu! \vi,L:ut in lii:^ cuurse 

3. Ketention beyond Involvement in the course: recall Qf or sayings 
time in relearning vocabulary and Information, concepts, 
methods, etc. 

4. Longitudinal transfer, extension, Incorporation, or 
elaboration: amount and kind of* application or continuity 

of use of content, concepts, methods, behavior, orientations, 
attitudes, etc. beyond involvement in the course 
The stated MACOS goals refer only to the first level. The . 
presumption, however, is that there will be other, more enduring payoffs 
such as those implied in the second and third levels and explicated in the 
fourth. In effect, acquisition, ^extension, and application (broadly) of 
knowledge, skills and/or attitudes, each conceived broadly also, may be 
viewed as a rough scaling of effectiveness that applies, potentially, to 
any course. It also applies to the effects or impact of the course on 
teachers. 

■ • 

Any evaluation study Interested in the third and fourth levels 
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of effect or 1«pact jwlth respect eHher to teacters or students must be 
longitudinal. It must extend at least a year beyond the presentation of 
the course, and preferably several years. 

mm is intended to be a process-oriented curriculum. Like 
most other te rns > process is ambiguous in that it has a number of simul- 
taneous referents. Children may learn hypothesis generating, exploring and 
testing processes, a family of behaviors referred to by v,arious people 
as inquiry nethods, research methodologies, etc. The means by which children 
learn or -are taught such processes cons titu** another family of behaviors - 
that generally include the teacher in one or more roles, functions, or 
capacities. What contingencies or functional relationships ( In Stake's 
tcniis'^} c:xist Itot-v.-rrn t-hcsc 1.v;o fumnior, of proccr.sc5. is nn ciripirical 
matter that could be examined by various evaluative as well as research 
studies. The nature of these relationships was, to be sure, one of the 
central" relationships of the conference on Learning by Discovery.^ 

If evaluation of the performance characteristics of MACOS as 
such is to yield more informaticn than is already available in the Hanley 
report, it should include consideration of at least the following: 

General Desion Requirements 

1. Repeated observation over a period of at least three j^ears of 
the same populations of students, teachers , schdols and school 
• systens. The need for follow-up with students is apparent 
from the Hai^ey report; the need for follow-»up with* teachers , 



? Stake, Robert E., op. cit. 
^ Shulman and Keislar, op. cit. 



jchcwls. and systeHs was apparent from tte InfottaTlRfornatlw 
gathered |jy this slu4y. ' - 

. 2, Explidt-attentidn to different prior, parallel , and subsequent 
^. , instructional progran^, organizations of instruction and 
.; . pedagogical orientations and styles to |^ich children studyinj^ 

KACOS «re .exposed.^ < ' . '1, X' 

. 3 . Expl i ci t attenti ga to the . processes of adopti on bf fhVcurri c- 
' ulum and the trainifig-and support -of teachers. . 
4. Documentation of the" variations in'-Joals and e)g3ect^tions ,of ' 

'teachers and school systems for the course* they vaiy, and- 
* Ihe explication bf congruences (again in Stake •Vterms) of \ 

• expeet^ions and o^ltcomss'f or of goals and e-xaniination content 
/ ' (in Scriven's tems^) is an Important output of an absolute 

evaluation. The,-e is a logical dl.lemma here; in that specific 
qoal criteria aro not osl.bl IMicc! in op-rncional .term^for 
HACOS, other than those iHipluTii in the content questionnaire. 

- m appropriate strategy, unaer such a circumstance, would be to 
measure as many outcoine variables as possible in order to examine 
differential relationships of goal expectancies and outcoines. 

5. Inclusion of a variety of outcome measures. I Th'ere.should be • 

* measures bf primary, secondary, and tertiary effect^ (again In ^ 
Scriven's terms) as implied In* the preceding points. But there 
should also, to the extent possible, be several forms of 

• measurement of the same variables, especially with respect to 
performance of children. Methods should include classroom 
observation, p^per and pencil tests, rating scales, interviews, 
and possibly the use bf kandardi zed. group tasks or projects. 

* Different methods will permit measurement of similar kinds of 
behavior'an<i competency under different task conditions. The 



behavior. S3 . .« 



conctuslwis one may draw about the atqulsHlon of, SAy^i 
Inquiry sMTIs, however defined, from responses to a paper 
and. pencil test may not be the -sas® as those drawn from the 
c^jervatlon ef the sarae' groups of d-.Hdren responding to a 
stinicturally- similar problem In an extended group problem 
' solving situation, of In an Interyl©'*- . 



^ 
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Specific Design and Ifethodo3oq1ca1 Con siderations 

The preceding requirements delineated general sets of independent 
and dependent variables, as well as general observation periods. It is 
not our intention to suggest specific designs and methodologies within- 
the frainework of absolute evaluation. It is appropriate. h««ver. to note 
some design and methodological considerations for National Science 
Foundation to include in choosing among subsequent courses of action, 
of which absolute evaluation Is one. 

1. Selection of schools, teachers and students for study. 
We assume that the general question of interest in an absolute 
evaluation is. what docs the course do with various kinds of students under 
v.riou^. tyr.. '. .-.n.-l cr.ndillo- of o6o?Uou flnd .-.ppl i .ati on? Th«rr or. 
essentially two kinds of comparisons of interest; within sub.iects (students, 
teachers) over time.and among treati^nts (different organizations of instruc 
tion. types of toachin?,. types of schools, etc). It is unlikely in the 
extren^ that there can be random assignn^nt of schools, teachers or students 
to tIACOS in anything ether than occasional local experiments. Furthermore. 
Inasmuch as the curriculum is only just beginning to be disseminated 
and adopted on a competitive basis nationally, any sample of schools and 
school systems chosen for inclusion in a study at time t (say, this 
calendar year) has an indeterminate relationship to all schools or systems. 

We suggest that an appropriate strategy is to try to define a 
current universe of using school systems, based on information available 
from the Regional Centers. EDO. and CDA, listing as well as possible the 
"systems and schools Reeling one or more conditions or levels of independent 
variables of interest. ' Then draw from this pool to the extent possible. 
This could be done within a single region, as presently constituted, with 

probably little practical loss of general Uy other than variations in 

05 



conditions and types of training for use of the course, and with consider- 
able gain in access and reduction In cost. We hypothesize .that the main 
loss would be in the face validity of results, based on the assumption 
that there is apt to be more variation within than betw^.en regions for 
most interesting variables related to the course. 

Another strategy is simply to construct clusters of schools and 
classrooms meeting eligibility criteria on various conditional variables. 
This is in effect the approach used by the EDC evaluators.^ 

2. Size of the study 

There is no simple way of designating the appropriate size of 
a study until one gets down to specific hypotheses and designs and 

poni':''tv:r C5t1]H:ih.-.. Thcnx: ^rc cJioicc":. :.-iiiDnci units of analyri:;. For 

some hypotheses schools may be the appropriate units, for others . t.1 ass monK 

and for others, students. When absolute evaluations are considered, it 

will be well to consider two additional factors in determining the size 

I 

of the study: 

a. follow-up with students in periods after their involvement 
in MACOS may be very costly if attempt is made to trace 
students associated with particular classrooms; 

b. in a longitudinal study one can expect the attrition of 
tea:hers as well as students to be high; in some schools 
attrition may run on the order of 15 percent a year for 
teachers and two to three times that or more for students. 



7 It was also, as far as we can tell, the method of the ERIE researchers 
whn are inves ti oatinq process curriculum installation problems, bse 
^y. ^ 'fcha?fs.' variables Affecting installation. In Andreas, Burton G 
et.al., Research Into Process C urricula . Eastern Regional Institute for Educa- 
rronT Syracuse, N.Y.» l-farch~T97irr' 
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We suggest that if absolute evaluations take the form of in-depth ana^ses, 
sudi aai |:hose implicit in Joyce's forthcoming report^, something on the 
order of' twenty to thirty teachers raay be quite sufficient. Othervise 
we would expect tli at numbers of teachers up to an order of magnitude 
greater than that will be needed. . • ^ 

Implementation ffechanisms 

There are several machanisms or vehicles for conducting a 
longitudinal evaluation. Each has its advantages and limitations. 
Mechanissr.3 that National Science Found=ition could consider include: 

1. providing a grant to an institution or agency (presumably 
a university or university-affiliated center or agency) to 
conduct ri specified ntudy 

2. providing] redundant or supplementary -grants to several such 
agencies either to replicate specific studies or^o conduct 
complementary studies 

3. setting up a m;\COS evaluation project agency to conduct studies - 

4. extending the capabilities of ooe or more of the present 
regional centers to include a research and evaluation function 

5. collaborating with other agencies (United States Office of 
Education; National Council of Social Studies ,ctc)to establish an 
evaluation project center, system or conmiission. 

We shall not undertake a detailed analysis of these alternatives, which 
apply to other evaluation approaches to be discussed below. The following 
remarks are intended simply to indicate some pros and cons of various 
mechanisms. 



^ Joyce, William W. HACOS: A Report from the Inner City. Interim report 
for Elementary Education Supplement, NCSS. (In press). 



or 



It Is Hkely that the imjst econcwiic logchanfsffl would be # 1 or 
I 2. Either of these has the advantage of providing a centralized control, 
of the entire process of specific studies They have potential limitations 
In'availablllty of technical people at Irregular psrieds during the academic 
year. In general, hovsfever, either option represents a minimum Investment 
. and v/ould thus seem to be appropriate vehicles If NSF envisions further., 
evaluation as a tem.inal activity. If. however, provision of continuous 
and useful feedback to participating or cooperating schools is Inqjortant 
to NSF, this is potentially the least effective mechanism unless special 

provisions are made. 

Option # 3 has the advantages of the first two, plus, the advantage 

of cnntinuouK concontration of rcsnura?s siid effort. It har. tho advantage 
of proviciing a focal point for feedback, and for -the coordination of further 
developniGnt work, especially with respect to methods of measurement and 
instrumentation. Especially if NSF sees the ultimate aim of further 
evaluation as producing information for more curriculum development projects, 

m 

an argument can be made that this option, or the fifth, are the most 

attractive mechanisms. The third option has the disadvantage, we think 

(but do not know) , of greater investment costs than the first or second 

options. It also duolicates the c?ngbilitics, or potential capabilities, of 

already existing centers and agencies elsewhere; It could alsr lack objectivity. 

The fourth option has a number of advantages. The regional centers 

have been the dissemination and training arms of the MACOS project. They have 

access to schools and to university resources. In the case of the ERIE center, 

'there hP*? ^'^en an active and impressive data gathering and feedback operation, 

even under the minimal funding available. 

The disadvantage that comes most readily to mind is the essential 

conflict of interest in co-locating a formal evaluation function 
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with a disseraination and training function. Regional centers are, after 
all. In the business of disseminating^ I4AC0S and training teachers. If., 
the essential aim of an absolute evaluation Is to delineate the various 
achlevfiJTBnts that the curriculum can have under various conditions, as 
Bruner has urged^ then appending an evaluation function to the centers seems 
appropriate. Otherv/isc, and with no imputations concerning the integrity 
and objectivity of regional center staffs.the evaluation function should be 
kept separate from the operational functions, in o~ur opinion. 

A combination of options # 3 and # 4 could conceivably counteract 
the conflict of interest problem. ^Even if option #4 were selected, it 
would be necessary to have a coordinating and directing office or agency 
to prnviclo ynifor;,! sprci fi c?iti ens nnd to ar.t^inv coniparal.nity of results. 
The inod.:l here mouU bo soir^lhiny like Bond and Dykstra's Cooperative 
Research Project in reading.^O though probably smaller in scale. 

The fifth option has a number of obvious advantages, especially 
if NSF sees^tself moving more broadly into, further curriculum development 
in the social studies, or into projects with a potential for institutional 
change. There are already those who are calling for large-scale, federally 
funded programs to manage research and development related to thinking 
skills and processes comparable to the Right to Read program. Goodlad 



9 Bruner, Oerome S. Some Elements of Discovery. ' Ch.VII in Shulman and 
Keislar. op. cit. , p. 113. . . ^ ^ 

10 Bond, Guy L., and Dykstra, Robert. Coordinating Renter for First Grade 
Reading iiistruction Programs. Final Report, Project No.X-OOl , Contrao No. 
OE-5-10-254, Feb., 1967. ERIC ED-013-714. , 

11 , ..nrfsteen Sara W. Critical Listeniaig and Thinking: A Recommended Goal 
for ru ^ '"search. JournaUofJMe^^ 

Vol.3 (1)". Fall, iQsgTTO'^sa. 
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recently called for broader, more coordinated efforts In curriculum Im- 
provement and refoniiJ2 Auhough this option is more germane to comparative 
or systemic evaluations than to absolute evaluations. It Is not altogether 
inappropriate for consideration here. Its disadvantages are easy to imagine. 
If not verify. They include cun^e^^omeness , problems, of inter-agency coordin- 
ation, and. perhaps, from the NSF point of view, abrogation of competitive 

interests and advantages. 

There are other conceivable nschanlsms, to be, sure.. We thi-nk 

these are the principal ones likely to be considered. 



Objectives and^ Payoffs 

Stake ended his now classical paper on the countenance of educa- 
tional evaluation v/it^ fivo questions that ought to be answered before undor- 

■Ukincru un..i cvaUu.liur.J^^ Four uf Ui. iivc apply to^^consi d.rafi on of 

absolute evaiuotion of MACUS. 

.1. Is it to be primarily descriptive / judginental or both? 

2 Is it "to emphasize the antecedent conditions, the transactions, 
or the outcomes alone, or a combination of these, or their 
functional contingencies?" 

3. Is it "to indicate the congruence between what is intended 
and what occurs?" 

4 Is it "intended" more to further the development of curricula 
or to help choose among available curricula? 

We have already suggested that absolute evaluation of MACOS would 

seem to relate mainly to adoption and implemWation decisions by schools. 

and to certain decisions concerning future cVriculum design and 

development. Realistically it would seem that the emphasis of an absolute 



^ Goodlad, John I., op. cit. 
Stake, Robert E. op. cit. 
. 1* Stake's other question-is this to be a comparative evaluation?- 
is answered in this discussion by definition. 
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evaluation of a curHculuin that does not have specific performance standards 
or objectives should be descriptive. It should probably emphasize antecedent t 
conditions and outconies if it is to serve the purpose of contributing to ^ 
adoption decisions --an urgent (^jective, given %im lags and other '^^ 
' considerations. OthenHse. it should probably en^hasize functional contin- * ' ; 
gencies among the three. This would yield information supporting both 
course improvenmnt strategies and decisions as well as, possibly and more 
remotely, further, curriculum design decisions. Indications of congruence? 
between intentions (goals) and wh^t occurs would seem to relate to decisions, 
concerning curriculum design, and to dissemination and installation methods 
and practices. 

It v/ould Bppnfn- that the p-iynffs for NST of one or another form • 
of absuluLo evaluation of TiACOS liiiige to an oxtent on how it views its role 
vis a' vis the curriculum. If it wants to assure that the curriculum whose 
developmsnt it has sponsored receives maximum support and assurance of 
effective and continued use, then an absolute evaluation approach is 
probably the best vehicle. If it sees its role as spawning further curric- 
ulum projects in the social studies, then absolute evaluation may or may 
not be the most productive or appropriate for obtaining necessary information. 
It may well not bear on the decisions NSF wants to make. In either case, 
there will of necessity be long lead times. They, can be reduced to some 
extent by interim reports. In the case of the first role, however, there 
must be special consideration of the feedback loops and feedback forms ahd 
formats between study and users. 
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Comparative Evaluations 
"The conclusion seeirs obligatory that ccaiparative evaluation, 

' M 15 ' 

mediated or not, is the inethod of choice for evaluation problens. ' 
Me started this stu^y with the presumption that cofi?)«rati ve evaluation was 
the next logical and necessary step In evaluation of MACOS. Hantey's 
formative evaluations included conipari son groups. Would not a useful 
further appraisal involve extension of the comparisons made In Hanl^y's 

studies? " ; 

The problem is, what are meaningful comparisons? The criterion 
of maaningfulness ought to be that the comparison leads to or is germane 
to decisions of general consequence. It is not clear in this Instance ^■ 
what such drcisions micjirt bc^. To adopt l'ifXC6 instoad of sen- olher'social 
studies curriculum? To launch another lorgL-scale development project? 
To modify the MACdS curriculum? To use more multi -media curricula? To 
develop more inter-disciplinary curricula? To develop more task or goal 
oriented curricula? Perhaps the real issue is metho?lolo^1cal . Can 
. meaningful" comparisons be made without actively manipulating some experi- 
mental variables? Scriven's engaging discussion of practical control 
group evaluation uses examples in which the evaluator has the power to 
manipulate some interesting independent variable. Hanley's comparisons 
were drawn from targets of opportunity, so to speak. Is it likely that the 
use of selected in NTivo contrasts can^ever be^ anything other than suggestive? 
This line of analysis quickly becomes an analog of the inquiry process 
students are being taught. The differences are the tiim scale, • . 



Scriven, Michael, op. cit. Scnvcn's position is, basically, that 
evaluation eventually is the judgment of worth or merit, ^^is pos tiom 
applicable to absolute evaluation, but it clearly gains in feasibility by 
comparative evaluation. 

6^ 
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tHe presumed need to nake practical decisions « and the mor^^^sp^fate 

relationships of those engaged In the process. ^^^^ r 

The basic fssue of cdn^arative evaluation In field settings is 

what really, is being compared. Hanley, et^. al,. , made coi^arisons of 

attitudes of children in f4AC0S and non-MACOS social studied courses* and of 

interview responses of MACOS and a limited number of non-MACOS teachers. 

Their othe-r independefrt variable was type of school system. They did not 

. . atten^t to conpare social studies programs as such, nor did they attempt to 

measure cognitive or behavioral outcome variables directly. Their findings 

of different reactions of children 'within MACOS treatirent groi'ps suggest 
■ * 

that there may well be pupi 1 -program-pedagogy interactions of importance. 

But the restricted ranqc of outcomo variables explored Jicrosr. trcatrtont 

c-roupi it difrJcu'lL Ic evjluulc inc b ujni f i caiufe of the findings 

v<itlrlh lite nACOS groups. 

Comparative evaluations ttre notoriously treacherous, especially if 

f . 

they cannot employ true experimantal control. Comparisons of intact groups 
observed in natural settings are subject to a variety of interpretational 
hazards when it comes to making attributive statements. Unfortunately , how- 
ever, con^lete or even constrained or restricted randomization of assignments 
is unfeasible on a large scale at this point'in time. (It might have been 
feasible during the develi)pmant stage of MACOS.) We shall discuss explanatory 
experimental courses of action later, but in the present context we do not 
believe true e^cperimental comparative* evaluation is a feasible model. Never- 
theless, MACOS has-^goals that are similar to the goals of other programs; 
or it seeks to stimulate skills j behavior, and attitudes, both in, teachers 
and students, that may or could occur with other programs. Only comparative 
V evaluation will help identify and clarify similarities and differences. . If • 

a comparative evaluation is to yield more information than is already avail- 
able in Hanley's study and if it, is not experimental, it should consider at 

k '1. ■ 

^ , — — ^ . ■ 

These points /are not at al.l .Intended, to denigrate Hanley's extraordinarily 
thorough, inform^ive, and creative formative studies. 




* : least the follo^ln^: ' 
-■'^^ General Design Requirements. . - 

Most are^ similar to those f^r absolute evaluation- ; 

' 1 Should include repeated observatlcms of the same student, 

' ' teacher, and school populations for at least three years. ; This 

: ' . ' is especially iiTHJoftant with respect to MACOS teachers, . ,g 

r face the task of learning a new curHcuaum. '| 
• Z. Should Include a variety of prior, concurrent and subsequent . 
organizations of Instruction, educational orientations, and 
demographic settings, both for- MACOS and non^MACOS groups. 

This obviously could easily result In a huge number of contlna- 
tlons of variables. If would probably be most appropriate to 
select clusters of schools and prograiM within geographically 
• and demographically similar areas. This would permit a.nunter 

• of specific sub- comari tons, in addition to asgrog.ite or,PS. 

■ ■ ■ " 3 There should be a variety of specifically selected comparison 

social studies programs, lhat is. rather than mcOS yersus 
hon-MACOS, comparisons should be made, say, for fourth, fifth, 
or sixth grade students in program A. program B. program C.etc. 
■• Comparison groups should include the best or most innovative 

• social studies programs availab>e. The risks of this are obvious, j 
and they serve to clarity the choices involved in comparative 
evaluations. One could, test several curricula, including 

■ - , MACOS, under contfolled and experimental conditions . using the 

best and most motivated teachers possible. This would lead to 
statements of high precision, but of uncertain application. Or 

one can do what is s,ugggsted here: seek performance measares 
for several contrasting curricula obtained under natural conditions. 
This w.ill lead to statements of uncertain precision and greater 
applicability. For example, it should lead to statements of the 

follovnng form: f^' 
^ X percentage of MACOS children in the lower (middlA 

upper) third of the class in inner city fifth grades 
shov/ed evidence in interviews of using a variety of 

O ^ . \ ^ 
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sources of tnfdmatlon In gathering data pertaining 
to <^rta1n questions, ccHi^ared viith y percentage of - - 
siianar* chliaren in W-MACOS program A (8,C, etc.). 
Such stateiients wiTl not explain anything; they witl sin«)ly 
depict what one may expect to find on the average under various 
conditions of utilization. One reason for suggesting that 
observations be made in relation to the same schools and teachers 
to the extent possible over a period of several years is to 
obtain a measure of the stability of such statements over time. 
Staten^nts of the sort suggested above can be an^lified to In-' 
elude more conditions, or to refer to more specific sub-groups. " 
- They would also apply to follow-up observations with samples of 

children in subsequent years. 
4. There should be a variety of outcome measures across programs. 
There ax;o thrcf> banc prohlcnr. here. One is that there should 
• ■ olif^cfVc; lions or ;:,j...oincs rcluLsd to eacli of thu» scvc-n lu\COl> 

gonU. %^\<^ron6 is that nnf ?11 social studlec: Trosrnms 
necessarily share the same goals, or would accept the same 
evidence or criteria of goal behavior. The third is that content 
and concepts differ among programs. The measurement and instru- 
mentation problem is central and probably poses the major dilemma 
■ and Choi ce for continuing evaluation of MACOS, whether absolute 
or comparative. 

There should be no serious problem in compiling or constructing 
generalized attitude scales and activities checklists for administration to 
students and teachers, regardless of program or. curri culum. Similarly, 
there are scales that observers, including teachers, can use to rate the 
extent to wh'ich- students demonstrate evidence of particular cognitive skills 
'pre^^'nably related .to inquiry processes. Inductive thinking, and so on. ^ 
There are classroom interaction checklists or scales that are independent 
.of course content, including those used in Hanley's studies. Se^s of 
' ndn-content specific items could be used to obtain measures of performance. 



65 



across programs. Some Hens or scales will vary in their application to - 
different courses and'c1asS6s; At least, -the categoHe^ of behaviors or 
psychological functions presumably measured by different f terns be ciore 
or less consistent with the' categories of behaviors, or functions developed - 
by this or that course, ^ut surely the reason tor making cojanarlsons Is 
to obtain some indication of what MACOS children, for example, are likely 
to do In relation to what non-MACOS children do on the same measures. ^ 

. Interviews could be deyeloppd io ascertain whether chiHren Who have 
had MACOS are more likely to give evidence of- having applied what they 
learned, or maintained an interest. germane to the course outside the 
classroom, than children in other social studies courses. There is. havever. 
no conv-incing way that we can see of directly comparing concept or kno'wlGdge 
attaind'Ht, u^^ Vi/.licm , r.nd r:tc.>n ti n,. f-cro.s coun^cs willioiit UuvA'iiKi^eqtn v- 
ale-nc.R of difficulty ^-in^/or functional cnuivi}cncc cf pfi:cticc. . ^• 
This argument does not apply t» such MACOS goals, however operation- 
alized, as encouraging "children to reflect upon their own experiences." or 
to "...legitimize the search. """^ Here it would seem to be precisely whether 
or not children in>1AC0S are likely to give more evidence of these (implied) 
behaviors than children in other social studies courses that is of interest 
By the same token, if there are generalized skills, attitudes, and behaviors 
or processes that are stimulated and promoted in other social studies 
projects, it would be reasonable to ask to what' extent they occur in MACOS 
children, intended or. othen^fise. 

X Choice of measures and instruments, be they paper and pencil tests. 



E'l'cr'ion Development Center. Man; A Course of Study Evaluation 
Strategies , op. cit., p. 12. 
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observer* ratings. Interviews with individuals or with groups , assignment 
of group projects, use of creative formats transcripts of class discussions . ^ 
or whatever, is inportant on ti^o counts. One is technical, the other cost. 
If there is the need for a variety of measures, and ^ believe there is. . 
one is faced with the decision either to administer what cou-ld amount to 
substantiaVbatteries to groups oN:hildreh and/or classes, or to administer 
san^les of itetos from different sets. The latter is £he better choice, 
since with the possible exception of certain follow-up observations of 
children, we presume the comparative questions of interest apply to groups, 
not individJals. To our knowledge, hwever, there are not sufficient item 
pools available from which to draw for measures of applicable cognitive 
or ).vnr-o .c"/ s.v..ncl piofn:-.:.. It world mc-,^^ry to dnv.lo..^ 

them, perhaps after tlie ^-HShinn of tho UatSnn.l A-crf,n-t Prcgrani a 
large and costly undertaking, or to patch together a battery of instruments 
for application at different points during the year and thereafter. The^ 
battery would probably have to, be limited in Scope to be at all attractive to 
participating schools, and to avoid extravagant administration, and processing 

costs. / ■ / 

The Choi ce , then, appSnrs to be between 1) a Targe scale instrunent 

developn^nt activity for measuring or observing development in children of , 

cognitive (including 1 magi nal ). emotional and behavioral skills^and orienta^ ^ 

tions related to social studies, and 2) the use of a limited set of instruments 

i 4 

of uncertain reliability . and adequacy. The first would perhaps be of most 
value in the field of elementary social studies in the long run. It is 
difficult..however. to see how a large scale effort would be justified simply 
to permit, initially, comparative evaluation of flACOS. The latter is . 
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ffeas#b1e. although H will su«ly yield the nore equivocal results.' . ^' 

fts a final con«Bnt, It should be apparent that the preceding " 
analyst* wiles primarily to the nteasuremenf of effects. on children. . . 
Heasu«s-^ outp«« ;jith respect to teat^er, . schools and co^nunlties seen, 
less prJbleWic. although here too theoretical and~>nethodological problens - 

can easily become severe and costly. 

.-- . "" ■ 

• Specific Dsslcinilcqui rements 

1. Selection of pro^ranis'or curricula. ^ . 

Given the kinds otf comparative state<aents we have suggested, it 
is desirable to d^Biw samples of each compari sonogram from 
known populations/but there probably is not enough available 
information to do that. It would probably be necessary to sample 
amonci and within r.chool distrir.ls on c|nonraphic and dcmbnrapnic. 

schools and teachers may be difficult, if not impossible , in ^som. 
cases. This approach does not preclude inclusion of cross-subject 
i^ariations in settings; e.g.. HACOS and some other curricullim , 
in conjunction with and not in conjunction with other process- 
oriented curricula such as Science-A Proce^ ,iP££gach.- It 
would be more appropriate to study that sort of interaction , 
separately, however, that is. in separate studies. 

2. Size of the study 

This will probably turn out to depend on the size of the 
differences one would like to detect, and on the extent to 
i ^ which it is deemed desirable and feasible to make observations 

in classrooms or othenvise to collect data during a school year. 
It is our opinion that detailed observations of processes or 
transactions are neither feasible nor necessary for the form of^ 
comparative evaluation discussc<l here. We have assumed that the 
relationships of primary interest are betweem input and output. 
Sample sizes also must be planned with attrition in mind. 
Attrition here, as with absolute evalutaion". refers to pupils, 
teachers, and programs. ' . . 
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Implementation tfechanlsws 
. There are no obvious differences in choices from those discussed 
earlier, with the following possible exceptions. If the study is to be 
very large (e.g., 1.000 teachers) , there may well be som advantage in 
establishing a MACOS evaluation project agency or collaborating witli other 
agencies. This would also be true if it were decided to pursue an extensive 
instrument development course. 

Objectives and Payoffs 

Results of comparative evaluations would seem to bear most directly 
on adoption and continuation decisions at the school district and. conceiv- 
ably, state levels They would a-l-so, logically, bear more directly on 
ciecisic.;,. .oncvr.ir.. fni..,- curriculun dc.irn .nd d.volc:pi..nt . D.pcndinn on 
the measures made and on the outcome., it is possible tli.at results v.-ould 
have a bearing on decisions about in-service training and support. 

The payoffs for NSF would appear to be principally the fulfillment 
of a complete development cycle and the development of further scientific 
knowledge about process-oriented curricula in the social studies. Results 
of a comparative evaluation may be useful for future decisions about 
curriculum improvement projects. As noted etrli^r, hawever. lead times will 
be long, even with annual reports of results. 

Should NSF elect to undertake major development of instrumentation 
ror measurement in the social studies, the payoff could be an interesting 
indirect form of course improvement regardless of curriculum. It was our 
observation, admittedly very limited, that the measurement and evaluation 
.of students' progress in MACOS, at least, was a source of major uncertainty 
or ambiguity to teachers, and to others, from one frame of reference. If 
instruments and guidelines were available, they could be of use. This is not 



In the least to suggest that fine teadiers are not fully capable of eval- y> 
uating their work with their students. They are, hov\rever, dealing with 
functions whifch, despite the technology of behavioral objectives, are not 
well understood or explicated. They are dealing with enormously complex 
patterns of behavior. Possibly aids in recognizing, classifying, and 

, TO 

evaluating them v/ould be qui ta welcome. 

We have taken the position that a true experimental comparison on 

• - y 

a large scale is not a feasible alternative. This in no way precludes the ' 
feasibility of one or more localized true experiments. We suggest, however, 
that while the internal validity of such experiments may be high, external 
validity is act to be very uncertain unless the local experiments are 

ropoalod v/ith a rr-nn:? of ?^ottinn^' tind po;.ul fili on?; . In that a-'-.c, hov.'i.vcr , 
it ib di i ViciJit tu m:.iy\^y'^^\l^J ciblu to iinplenieiit comparison currlcuU 
satisfdctojily. Nevertheless, v/g do not wish to suggest tliat we consider 
true experi mental comparative evaluations impossible. True experimsnts 
hold the greatest hope for yielding explanatory statements or hypotheses. 
We shall return to suggestions for experimental research and evaluation 
running parallel to a larger study following the next section (see Explan- 
atory Studies, p. 70-71). 



The recently published booiOet on evaluation techniques for 14AC0S 
(jjglL L ^P-HrgiULl- study _Evfll udtion S trategi es ) should be of great help to 
many toa^cliers in Ws' respect. 
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Systemic Evaluations 
Me have grouped under the heading of systemk evaluations studies 
whose focus is one or more of the various models, channels, nechanisns, 
procedures, arrangements, or organizational consequences of the dissemina- 
tion, adoption, and support of MACOS. There is no sharp boundary betvveen 
systemic evaluation, tlius conceived, and curriculum evaluations, absolute 
or comparative. Ultimate cHterion measures of performance will, theoreti- 
crjG, be the same in all cases. Intermediate criteria of effectiveness 
and efficiency will differ, however. Consider the difference between the 

following questions: 

How effective is the curriculum in "(invelop[inq] in youngsters 

•l&W crr.cL-:.. U ll.i. u. U.ul Ltuininy Mroyram in developing 
in teachers skill to promote the question-posing process in children? 

. In the first case an answer could be obtained with no reference 
to variations in training programs or models if care were taken to guard 
against or minimize sampling biases. In the second case, an answer could 
not be obtained without specifying measurements of training programs. 

There are an infinite variety of evaluative questions that can be 
examined with reference to MACOS. Some or all in one way or another overlap 
with or impinge on areas of investigation already well established in 
different centers and universities. Our limited field observations left 
us with the conviction that three crucial functions in adoption and 
implementation of the course are selection, training, and quality control. 



Educational Development Center, :i^2]}jjL^Jm^^L^^''^'^ Evaluation . 
Strategies , op. cit., p. 12. 
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There were wide variations In all functions, ranging from no formal pro- 
cesses to elaborate and continuous activities. In some cases in which 
there had been apparently very good training for very capable teachers, 
the teachers nevertheless indicated specific tasks for which they felt 
inadequately prepared. Examples of variations in the other two functions 
have been mentioned througfjout this report. Our argument is that 

1. the processes and arrangements by v/hich the Implementation 
of the MACOS curriculum comes about and is supported can be 
the subject of evaluation; and, 

2. the use of MACOS in different settings can have systemic 
consequences that can be the subject of evaluation; and, 

3. either or both subjects of evtiluDtion may be as v/orthv/hi Ic 
curri cul uiii. 

It is clear, as v/e have noted before, that systemic evaluations 
are not independent of curriculum evaluations. Indeed, a truly compre- 
hensive curriculum evaluation v/ould include all the components and dimensions 
of systemic evaluation. It would also probably be impossible to do in 
a form, time frame and budgst that would provide widely useable results. 

One of the basic problems in systemic evaluation, especially in 
appraising training and support models,. is similar to the problem of 
evaluation of MACOS: what are appropriate criterion measures or dependent 
variables? what are the parameters of different training and support models? 
what are the interactions or differential sensitivities of methods and 
teachers? I4ethodologi cal ly , systemic evaluations will perforce differ 
depending on v/hether one focusses on input (dissemination, adoption. 
Installation, training, support) or output (impacts or effects on teachers, 
schools, systems, communities, colleges, state agencies, etc), or on both. 



It Is worth noting that^in a way, input evaluation, in the above 
sense. Is the task of identifying convergences. Its aim is to establish 
relationships that hold hope of maxiinizing successful use of the curriculum. 
Output systeiiiic evaluation is the task of identifying divergences— the various 
consequences of adoption and utilization of the curriculum. Its aim is to 
establish possibilities, anticipated or planned, and unanticipated. 

There is no question about the feasibility of evaluating the 
diffusion of MACOS, and its short term and long term systemic impacts. 
Periodic surveys would be one vehiele for accomplishing this. There are 
,many questions, however, about the feasibility of evaluating the effective- 
ness of various training and support niochanisms and approaches. Such 
. study -^.-c---:. ih-: s proM-M • • ir, r,'? cri l-. rk-. jti^'j'i'j 

pf f^^ri-j uf.ipfKsc A+" th'.: cc'jrcc '..'it!. stuiJ^.ita. uuL i I iins liie ticdi ti ona I 
difficulties of measuronisnt or observation of adults— teachers , teachers 
of teachers, support personnel, supervisors, etc. And it fac^^ the further 



problem of diversity of training and support programs (potehtr^lly) milike 
the curriculum which is a fixed and constant package by comparison. l\ 
must also take irito account the resources and orientations of different 



Development Associates, unless National Science Foundation or some other 
agency continues to support, training programs through the regional centers. 

As with evaluation of the cur^^iculum, it would be necessary to " 
measure a variety of dependent variables., and a variety of methods of 
observation, including interviews, checklists, and classroom observation. 
Evaluation of MACOS traini'ng models makes most sense, of course, only if 
there continue to be training programs for the curriculum. If, for 
example, tW regional "centers continued to provide trainingj'nsti tutes , an 




schools, systems, and communities, as well as of the publisher, Curriculum 



evaluatlorv of the different models and approacl^es enployed by the centers 
could" be roade.^^ 

The mechanisns or options for undertaking systemic evaluation are 
mst likely those discussed In connection with absolute evaluation. Since 
cross-nradel study should employ the same measures, or samples of items from 
the san© pool of items (as with curriculum evaluation) , and since there are 
different theoretical orientations among the shaplemgnters of different 
training models, it v/ould be appropriate to nave the first phase of a 
systemic evaluation concerned with training start with a design conference. 
At such a confei-^nce, trainers as well as measui'^ment and evaluation special- 
ists, teachers, and supervisory personnel should have the task of identifying 
thG var1a!n'jJ.i n '•-i^uirrf; , v.wd crilerici thnt should ho inclih-!'- d or cori^i c'crcd. 

What is the payoff of systeinic evaluation for NSF? As PVAr» it 
depends on the goals and roles that the Foundation sets for itself. If a 
goal is course improvenient in the social studies through 'further curriculum ' 
development projects, it. would seem that only output systemic evaluation 
would be germane (evaluation of impacts and effects of the curriculum 
on schools and, hopefully, teacher prppaj^ation institutions). If a goal is 
course improvement through system improvement, then a broader systemic . 
evaluation is germane. 



' It is possible to set up systemic evaluation projects that include 
training evaluation. Thus, for several different training programs, one could 
set up several CHAS analysis projects in a number of school districts, 
coupled with follow-up empirical evaluation. See Hcrfi.sett, Irving, et.al., 
A f4odel for Analyzing Curriculum Materials and Classroom Transactions, 
ch. 8 in Doroth)<s„McClure Fraser, ed. , S£cj_al Stud ios Curriculum Development ; 
Prospects and Problems . 39th Yearbook, NCSSTl959. 



: Explanatory Studies; A Final Note 

Th^re are many explanatory studies' that can and should be done 
with the I^COS eurrlculum. . These -are the traditional province of expe^i- 
mental research. It Is likely that such studies will appear Increasingly, 
especially in the form of graduate dissertations. They will, find their 
way uncertainly back into the public domain in annual reviews of research 
(e.g., in the National Council of Social Studies magazine Social Studies) 
and become at least incorporated into professional and technical writings. 
Since graduate dissertations are undoubtedly the most e(fonomic form of 
. potentially good research, it is strongly suggested th-at National Science 
Foundation consider subsidizing some each year in specific evaluative 
smpoTt of rurrir.uluni projo-l«.. ni.Th r,:, i:Arni-, it if dn::s not :ilvciu\y do so?\ 

if ^ A 

Sina^'a nui'f)U^ oT qujsuions about KACUS and oLucr curricula are ansv.'urable 
only on the bdbis of WGll-controned research studies that need not be 
national in scope^^support for a number of pertinent res^Vch dissertations 
could be a programmatic alternative of considerable merit. The utility 



"■^ Lest it be supposed that doctoral dissertations , for example, are unlikely 
to contribute to theory, knowledge, and practice, one should note the 
central ity of Griffin's graduate thesis in M^tcalf's chapter. Research on 
Teachinq the Social Studies (in Gage, N.L.. ed. , HandbooLo f Resea rchj)n 
Sinl Chicago: Rand hcNally, 1963. ch,17); or tHe number of dissertation 
references in the Review of Research in Social Studibs, 19o7i yo'-^^j;^» . 
NO e! Oct. 1968, by Cox, C. Benjamin, et al- Sixty-two of the eighty-three 
references were to the Dissertation Abstracts . v 

22 The adequacy of the experiment will always be contestable, especially 
if it is small and significant differences do not emerge. This can be addressed 
to som^ extent by asking the dissertation writer to note size of the 
differences he could expect to detect at some level of Pr^f ^^^^^J » fiyen 
the necessary parameters of his experiment. The issue with much individual 
research is not what maqnitude of differences should be detected, but what 
magnitude can be detected with the resources available. 
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of this approach would be enhanced if NSf had or could assemble a pool of 
interesting hypotheses related to HACOS specifically/or to social studies 
curriculun, iinprovement generally, to use as a gulde in seeking and/or 
funding specific thesis projects." . ■ • 

TheBe are obvio^ly other vehicles, for conducting research into 
►UVCOS. and we do not n^an to eKclude consideration of the». The essential 
point is 'that we hav; tried to make a,distin4tion between further course 
evaluation and ,xperi*«'ntaV rejearch studies of n,any of the ^ultitude^of 
theoretical problens'^and issues related to the course and its use. The 
iuTinction. Of course, blurs and disappears as evaluative effort becomes 
,/>re concerned with explaining relationships, differences, associations, 
ckc. , v.rn a-, d-j1im-.-.t1n(j tlinii. 



'SUMMARY AND CONCLUDING COt^tENTS 



In the preceding sections we have .tried to delineate Issues and 
altematlvfis for further evaluation of MACOS. The purpose has been to 
suggest to the National Science Foundation some bases for choosing among 
alternatives. There are many choices, depending on one's concept of eval- 
uation and on one's purposes in evaluating. Wo have analyzed alternatives 
from the point of viev/ of steps the NSF might take. We do not suggest 
all possibilities have been exhausted, even from that perspective, although 
we think most major ones have been considered. If the problem v/ej-e 
approached from the point of view of a local school , a school system, a 
state aciency or rn ind-jpond- nt rr'r.^ardir.T^for orrii.ijjlG ,oti;er a1 loriirili vc-r. 
or rulionalcis i;iiy!it hii appropruLt'. Ue have nnt.bcieti able to provide 
a concise definition of trade-offs among alicrnatives, certainly not in a 
quant i tat* ve sense. We have, however, at lecjt provided a start at depicting 
what trade-offs may be. I 

The main decision criteria we have suggested for choosing among 
alternative strategies have been: 1) the kinds of decisions to which 
different sorts of evaluation information seem to be related; and 2) the 
goals and roles N^F may have or want to have. It has been obvious that 
neither criterion uniquely differentiates alternatives. The first criterion 
especially does not differentiate alternatives, at least at the global 
and unspecified level at which we have considered it. 

We have not considered costs as a criterion for choosing among 
oltemativp strategies. It is apparent that costs will depend mainly on 
tlie size and depth of study made within different alternatives (absolute, 
comparative and systemic evaluations), and on\the mechanism chosen for 



tmplementation. They will also depend on the extent to which developinent 
of new Instrumentation Is undertaken. 

Our position has been that It will be possible to make meaningful 
and defensible evaluative statements using inethods and instruments currently 
available or reasonably easily developed. This is not at all the same as 
saying there is no need for development of better instrumentation. There is 
especially need for better instrumentation with respect to measuring higher 
order cognitive structures and skills, such as hypothesis seeking and 
problem solving, or social learning, or affective orienta^Tms and skills^ 
A recent restatemant of the need was made by Coffman: 

"There is a.mountsin of unfinished buniness in this area 
flcr.l- d-v.:'lo; ;■'(.'• 1] if \r-- arr to itljvIc!'.: r.o;\:l.!.i li'Ovr: Ihr.ii a 

(.-t.r ;! vi'-'..' n: r: Ti c' tliC oul\-u'^ c:- c'-' '■'^^'■\-^' „\ 

through tusts of cucjniLivc; skills and subject laatter knowlcuye. 

Nonetheless, further evaluation of MACOS is possibl-e without undertaking 
a major instrument development effort. 

Our basic position is that there Is no one maasure of effective- 
ness of MACOS. Each of the course's goals can be given operational defini- 
tions; or criteria for classifying and recording the occurrence of goal 
related behavior can be developed. The goals may be the development of 
• processes, as well as products. The goals may include the development or 
enhancement of dispositions, orientations, or attitudes. They may refer 
th children or to teachers. In any case one can establish cri teria' for 
identifying appropriate behavior, and conditions for observing it. can be 
set up- interview?, classroom projects, classroom ratings, paper and pencil 



T Coffman. William E. Concepts of Achievement -nd Proficiency. Pro- 
ceedings of the 1%9 Invitationral Conference on lesting-Problems . Educar 
tional Testing Service. November 1, 1969, 3-11. 



